{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CTR预估(1)\n",
    "\n",
    "资料&&代码整理by[@寒小阳](https://blog.csdn.net/han_xiaoyang)(hanxiaoyang.ml@gmail.com)\n",
    "\n",
    "reference：\n",
    "* [《广告点击率预估是怎么回事？》](https://zhuanlan.zhihu.com/p/23499698)\n",
    "* [从ctr预估问题看看f(x)设计—DNN篇](https://zhuanlan.zhihu.com/p/28202287)\n",
    "* [Atomu2014 product_nets](https://github.com/Atomu2014/product-nets)\n",
    "\n",
    "关于CTR预估的背景推荐大家看欧阳辰老师在知乎的文章[《广告点击率预估是怎么回事？》](https://zhuanlan.zhihu.com/p/23499698)，感谢欧阳辰老师并在这里做一点小小的摘抄。\n",
    "\n",
    ">点击率预估是广告技术的核心算法之一，它是很多广告算法工程师喜爱的战场。一直想介绍一下点击率预估，但是涉及公式和模型理论太多，怕说不清楚，读者也不明白。所以，这段时间花了一些时间整理点击率预估的知识，希望在尽量不使用数据公式的情况下，把大道理讲清楚，给一些不愿意看公式的同学一个Cook Book。\n",
    "\n",
    "> ### 点击率预测是什么？\n",
    "\n",
    "> * 点击率预测是对每次广告的点击情况做出预测，可以判定这次为点击或不点击，也可以给出点击的概率，有时也称作pClick。\n",
    "\n",
    "> ### 点击率预测和推荐算法的不同？\n",
    "\n",
    "> * 广告中点击率预估需要给出精准的点击概率，A点击率0.3%  , B点击率0.13%等，需要结合出价用于排序使用；推荐算法很多时候只需要得出一个最优的次序A>B>C即可；\n",
    "\n",
    "> ### 搜索和非搜索广告点击率预测的区别\n",
    "\n",
    "> * 搜索中有强搜索信号-“查询词(Query)”，查询词和广告内容的匹配程度很大程度影响了点击概率； 点击率也高，PC搜索能到达百分之几的点击率。\n",
    "\n",
    "> * 非搜索广告（例如展示广告，信息流广告），点击率的计算很多来源于用户的兴趣和广告特征，上下文环境；移动信息流广告的屏幕比较大，用户关注度也比较集中，好位置也能到百分之几的点击率。对于很多文章底部的广告，点击率非常低，用户关注度也不高，常常是千分之几，甚至更低；\n",
    "\n",
    "> ### 如何衡量点击率预测的准确性？\n",
    "\n",
    "> AUC是常常被用于衡量点击率预估的准确性的方法；理解AUC之前，需要理解一下Precision/Recall；对于一个分类器，我们通常将结果分为：TP,TN,FP,FN。\n",
    "> ![](https://pic4.zhimg.com/80/v2-1641631d510e3c660c208780a0b9d11e_hd.jpg)\n",
    "\n",
    "\n",
    "> 本来用Precision=TP/(TP+FP)，Recall=TP/P，也可以用于评估点击率算法的好坏，毕竟这是一种监督学习，每一次预测都有正确答案。但是，这种方法对于测试数据样本的依赖性非常大，稍微不同的测试数据集合，结果差异非常大。那么，既然无法使用简单的单点Precision/Recall来描述，我们可以考虑使用一系列的点来描述准确性。做法如下：\n",
    "\n",
    "> * 找到一系列的测试数据，点击率预估分别会对每个测试数据给出点击/不点击，和Confidence Score。\n",
    "\n",
    "> * 按照给出的Score进行排序，那么考虑如果将Score作为一个Thresholds的话，考虑这个时候所有数据的 TP Rate 和 FP Rate； 当Thresholds分数非常高时，例如0.9，TP数很小，NP数很大，因此TP率不会太高； \n",
    "> ![](https://pic2.zhimg.com/80/v2-77e1e16ee58697a316cfe2728be86efe_hd.jpg)\n",
    "> ![](https://pic2.zhimg.com/80/v2-10666128633da6ea072a4c87f21d6bdf_hd.jpg)\n",
    "> ![](https://pic3.zhimg.com/80/v2-d70746453ced3e20a04f297169bd12bf_hd.jpg)\n",
    "> * 当选用不同Threshold时候，画出来的ROC曲线，以及下方AUC面积\n",
    "> * 我们计算这个曲线下面的面积就是所谓的AUC值；AUC值越大，预测约准确。\n",
    "\n",
    "\n",
    "> ### 为什么要使用AUC曲线\n",
    "\n",
    "> 既然已经这么多评价标准，为什么还要使用ROC和AUC呢？因为ROC曲线有个很好的特性：当测试集中的正负样本的分布变化的时候，ROC曲线能够保持不变。在实际的数据集中经常会出现类不平衡（class imbalance）现象，即负样本比正样本多很多（或者相反），而且测试数据中的正负样本的分布也可能随着时间变化。AUC对样本的比例变化有一定的容忍性。AUC的值通常在0.6-0.85之间。\n",
    "\n",
    "\n",
    "> ### 如何来进行点击率预测？\n",
    "\n",
    "> 点击率预测可以考虑为一个黑盒，输入一堆信号，输出点击的概率。这些信号就包括如下信号\n",
    "\n",
    "> * **广告**：历史点击率，文字，格式，图片等等\n",
    "> * **环境**：手机型号，时间媒体，位置，尺寸，曝光时间，网络IP，上网方式，代理等\n",
    "> * **用户**：基础属性（男女，年龄等），兴趣属性（游戏，旅游等），历史浏览，点击行为，电商行为\n",
    "> * **信号的粒度**：\n",
    "> `Low Level : 数据来自一些原始访问行为的记录，例如用户是否点击过Landing Page，流量IP等。这些特征可以用于粗选，模型简单，`\n",
    "> `High Level: 特征来自一些可解释的数据，例如兴趣标签，性别等`\n",
    "\n",
    "\n",
    "> * **特征编码Feature Encoding：**\n",
    "\n",
    "> `特征离散化：把连续的数字，变成离散化，例如温度值可以办成多个温度区间。`\n",
    "\n",
    "> `特征交叉： 把多个特征进行叫交叉的出的值，用于训练，这种值可以表示一些非线性的关系。例如，点击率预估中应用最多的就是广告跟用户的交叉特征、广告跟性别的交叉特征，广告跟年龄的交叉特征，广告跟手机平台的交叉特征，广告跟地域的交叉特征等等。`\n",
    "\n",
    "> * **特征选取（Feature Selection）：**\n",
    "\n",
    "> `特征选择就是选择那些靠谱的Feature，去掉冗余的Feature，对于搜索广告Query和广告的匹配程度很关键；对于展示广告，广告本身的历史表现，往往是最重要的Feature。`\n",
    "\n",
    "> * **独热编码（One-Hot encoding）**\n",
    "\n",
    "```假设有三组特征，分别表示年龄，城市，设备；\n",
    "\n",
    "[\"男\", \"女\"]\n",
    "\n",
    "[\"北京\", \"上海\", \"广州\"]\n",
    "\n",
    "[\"苹果\", \"小米\", \"华为\", \"微软\"]\n",
    "\n",
    "传统变化： 对每一组特征，使用枚举类型，从0开始；\n",
    "\n",
    "[\"男“，”上海“，”小米“]=[ 0,1,1]\n",
    "\n",
    "[\"女“，”北京“，”苹果“] =[1,0,0]\n",
    "\n",
    "传统变化后的数据不是连续的，而是随机分配的，不容易应用在分类器中。\n",
    "\n",
    " 热独编码是一种经典编码，是使用N位状态寄存器来对N个状态进行编码，每个状态都由他独立的寄存器位，并且在任意时候，其中只有一位有效。\n",
    "\n",
    "[\"男“，”上海“，”小米“]=[ 1,0,0,1,0,0,1,0,0]\n",
    "\n",
    "[\"女“，”北京“，”苹果“] =[0,1,1,0,0,1,0,0,0]\n",
    "\n",
    "经过热独编码，数据会变成稀疏的，方便分类器处理。\n",
    "```\n",
    "\n",
    "> ### 点击率预估整体过程：\n",
    "\n",
    "> 三个基本过程：特征工程，模型训练，线上服务\n",
    "\n",
    "> ![](https://pic3.zhimg.com/80/v2-a238723a7c09cd540c3c874f9a4777d2_hd.jpg)\n",
    "\n",
    "> * 特征工程：准备各种特征，编码，去掉冗余特征（用PCA等）\n",
    "\n",
    "> * 模型训练：选定训练，测试等数据集，计算AUC，如果AUC有提升，通常可以在进一步在线上分流实验。\n",
    "\n",
    "> * 线上服务：线上服务，需要实时计算CTR，实时计算相关特征和利用模型计算CTR，对于不同来源的CTR，可能需要一个Calibration的服务。\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 用tensorflow构建各种模型完成ctr预估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 5:1 9:1 357943:1 445905:1 446144:1 446294:1 450548:1 479123:1 491626:1 491634:1 491641:1 491644:1 491648:1 491668:1 491700:1 491709:1\r\n",
      "0 5:1 9:1 403405:1 445920:1 446287:1 446293:1 452727:1 490863:1 491629:1 491637:1 491639:1 491645:1 491659:1 491668:1 491698:1 491708:1\r\n",
      "0 5:1 9:1 414259:1 445897:1 446049:1 446293:1 454694:1 491595:1 491625:1 491633:1 491639:1 491646:1 491660:1 491668:1 491674:1 491708:1\r\n",
      "0 5:1 9:1 430407:1 445892:1 446063:1 446294:1 448775:1 476818:1 491625:1 491633:1 491641:1 491644:1 491660:1 491668:1 491700:1 491709:1\r\n",
      "0 5:1 9:1 140858:1 445908:1 446177:1 446293:1 449140:1 490778:1 491626:1 491634:1 491641:1 491645:1 491648:1 491668:1 491700:1 491708:1\r\n"
     ]
    }
   ],
   "source": [
    "!head -5 ./data/train.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:other\t0\r\n",
      "0:0\t1\r\n",
      "0:1\t2\r\n",
      "0:2\t3\r\n",
      "0:3\t4\r\n",
      "0:4\t5\r\n",
      "0:5\t6\r\n",
      "0:6\t7\r\n",
      "1:other\t8\r\n",
      "1:00\t9\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 ./data/featindex.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "import cPickle as pkl\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from scipy.sparse import coo_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "field sizes: [25, 445852, 36, 371, 4, 11328, 33995, 12, 7, 5, 4, 20, 2, 38, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8]\n"
     ]
    }
   ],
   "source": [
    "# 读取数据，统计基本的信息，field等\n",
    "DTYPE = tf.float32\n",
    "\n",
    "FIELD_SIZES = [0] * 26\n",
    "with open('./data/featindex.txt') as fin:\n",
    "    for line in fin:\n",
    "        line = line.strip().split(':')\n",
    "        if len(line) > 1:\n",
    "            f = int(line[0]) - 1\n",
    "            FIELD_SIZES[f] += 1\n",
    "print('field sizes:', FIELD_SIZES)\n",
    "FIELD_OFFSETS = [sum(FIELD_SIZES[:i]) for i in range(len(FIELD_SIZES))]\n",
    "INPUT_DIM = sum(FIELD_SIZES)\n",
    "OUTPUT_DIM = 1\n",
    "STDDEV = 1e-3\n",
    "MINVAL = -1e-3\n",
    "MAXVAL = 1e-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 读取libsvm格式数据成稀疏矩阵形式\n",
    "# 0 5:1 9:1 140858:1 445908:1 446177:1 446293:1 449140:1 490778:1 491626:1 491634:1 491641:1 491645:1 491648:1 491668:1 491700:1 491708:1\n",
    "def read_data(file_name):\n",
    "    X = []\n",
    "    D = []\n",
    "    y = []\n",
    "    with open(file_name) as fin:\n",
    "        for line in fin:\n",
    "            fields = line.strip().split()\n",
    "            y_i = int(fields[0])\n",
    "            X_i = [int(x.split(':')[0]) for x in fields[1:]]\n",
    "            D_i = [int(x.split(':')[1]) for x in fields[1:]]\n",
    "            y.append(y_i)\n",
    "            X.append(X_i)\n",
    "            D.append(D_i)\n",
    "    y = np.reshape(np.array(y), [-1])\n",
    "    X = libsvm_2_coo(zip(X, D), (len(X), INPUT_DIM)).tocsr()\n",
    "    return X, y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 数据乱序\n",
    "def shuffle(data):\n",
    "    X, y = data\n",
    "    ind = np.arange(X.shape[0])\n",
    "    for i in range(7):\n",
    "        np.random.shuffle(ind)\n",
    "    return X[ind], y[ind]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 工具函数，libsvm格式转成coo稀疏存储格式\n",
    "def libsvm_2_coo(libsvm_data, shape):\n",
    "    coo_rows = []\n",
    "    coo_cols = []\n",
    "    coo_data = []\n",
    "    n = 0\n",
    "    for x, d in libsvm_data:\n",
    "        coo_rows.extend([n] * len(x))\n",
    "        coo_cols.extend(x)\n",
    "        coo_data.extend(d)\n",
    "        n += 1\n",
    "    coo_rows = np.array(coo_rows)\n",
    "    coo_cols = np.array(coo_cols)\n",
    "    coo_data = np.array(coo_data)\n",
    "    return coo_matrix((coo_data, (coo_rows, coo_cols)), shape=shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# csr转成输入格式\n",
    "def csr_2_input(csr_mat):\n",
    "    if not isinstance(csr_mat, list):\n",
    "        coo_mat = csr_mat.tocoo()\n",
    "        indices = np.vstack((coo_mat.row, coo_mat.col)).transpose()\n",
    "        values = csr_mat.data\n",
    "        shape = csr_mat.shape\n",
    "        return indices, values, shape\n",
    "    else:\n",
    "        inputs = []\n",
    "        for csr_i in csr_mat:\n",
    "            inputs.append(csr_2_input(csr_i))\n",
    "        return inputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 数据切片\n",
    "def slice(csr_data, start=0, size=-1):\n",
    "    if not isinstance(csr_data[0], list):\n",
    "        if size == -1 or start + size >= csr_data[0].shape[0]:\n",
    "            slc_data = csr_data[0][start:]\n",
    "            slc_labels = csr_data[1][start:]\n",
    "        else:\n",
    "            slc_data = csr_data[0][start:start + size]\n",
    "            slc_labels = csr_data[1][start:start + size]\n",
    "    else:\n",
    "        if size == -1 or start + size >= csr_data[0][0].shape[0]:\n",
    "            slc_data = []\n",
    "            for d_i in csr_data[0]:\n",
    "                slc_data.append(d_i[start:])\n",
    "            slc_labels = csr_data[1][start:]\n",
    "        else:\n",
    "            slc_data = []\n",
    "            for d_i in csr_data[0]:\n",
    "                slc_data.append(d_i[start:start + size])\n",
    "            slc_labels = csr_data[1][start:start + size]\n",
    "    return csr_2_input(slc_data), slc_labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 数据切分\n",
    "def split_data(data, skip_empty=True):\n",
    "    fields = []\n",
    "    for i in range(len(FIELD_OFFSETS) - 1):\n",
    "        start_ind = FIELD_OFFSETS[i]\n",
    "        end_ind = FIELD_OFFSETS[i + 1]\n",
    "        if skip_empty and start_ind == end_ind:\n",
    "            continue\n",
    "        field_i = data[0][:, start_ind:end_ind]\n",
    "        fields.append(field_i)\n",
    "    fields.append(data[0][:, FIELD_OFFSETS[-1]:])\n",
    "    return fields, data[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 在tensorflow中初始化各种参数变量\n",
    "def init_var_map(init_vars, init_path=None):\n",
    "    if init_path is not None:\n",
    "        load_var_map = pkl.load(open(init_path, 'rb'))\n",
    "        print('load variable map from', init_path, load_var_map.keys())\n",
    "    var_map = {}\n",
    "    for var_name, var_shape, init_method, dtype in init_vars:\n",
    "        if init_method == 'zero':\n",
    "            var_map[var_name] = tf.Variable(tf.zeros(var_shape, dtype=dtype), name=var_name, dtype=dtype)\n",
    "        elif init_method == 'one':\n",
    "            var_map[var_name] = tf.Variable(tf.ones(var_shape, dtype=dtype), name=var_name, dtype=dtype)\n",
    "        elif init_method == 'normal':\n",
    "            var_map[var_name] = tf.Variable(tf.random_normal(var_shape, mean=0.0, stddev=STDDEV, dtype=dtype),\n",
    "                                            name=var_name, dtype=dtype)\n",
    "        elif init_method == 'tnormal':\n",
    "            var_map[var_name] = tf.Variable(tf.truncated_normal(var_shape, mean=0.0, stddev=STDDEV, dtype=dtype),\n",
    "                                            name=var_name, dtype=dtype)\n",
    "        elif init_method == 'uniform':\n",
    "            var_map[var_name] = tf.Variable(tf.random_uniform(var_shape, minval=MINVAL, maxval=MAXVAL, dtype=dtype),\n",
    "                                            name=var_name, dtype=dtype)\n",
    "        elif init_method == 'xavier':\n",
    "            maxval = np.sqrt(6. / np.sum(var_shape))\n",
    "            minval = -maxval\n",
    "            var_map[var_name] = tf.Variable(tf.random_uniform(var_shape, minval=minval, maxval=maxval, dtype=dtype),\n",
    "                                            name=var_name, dtype=dtype)\n",
    "        elif isinstance(init_method, int) or isinstance(init_method, float):\n",
    "            var_map[var_name] = tf.Variable(tf.ones(var_shape, dtype=dtype) * init_method, name=var_name, dtype=dtype)\n",
    "        elif init_method in load_var_map:\n",
    "            if load_var_map[init_method].shape == tuple(var_shape):\n",
    "                var_map[var_name] = tf.Variable(load_var_map[init_method], name=var_name, dtype=dtype)\n",
    "            else:\n",
    "                print('BadParam: init method', init_method, 'shape', var_shape, load_var_map[init_method].shape)\n",
    "        else:\n",
    "            print('BadParam: init method', init_method)\n",
    "    return var_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 不同的激活函数选择\n",
    "def activate(weights, activation_function):\n",
    "    if activation_function == 'sigmoid':\n",
    "        return tf.nn.sigmoid(weights)\n",
    "    elif activation_function == 'softmax':\n",
    "        return tf.nn.softmax(weights)\n",
    "    elif activation_function == 'relu':\n",
    "        return tf.nn.relu(weights)\n",
    "    elif activation_function == 'tanh':\n",
    "        return tf.nn.tanh(weights)\n",
    "    elif activation_function == 'elu':\n",
    "        return tf.nn.elu(weights)\n",
    "    elif activation_function == 'none':\n",
    "        return weights\n",
    "    else:\n",
    "        return weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 不同的优化器选择\n",
    "def get_optimizer(opt_algo, learning_rate, loss):\n",
    "    if opt_algo == 'adaldeta':\n",
    "        return tf.train.AdadeltaOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'adagrad':\n",
    "        return tf.train.AdagradOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'adam':\n",
    "        return tf.train.AdamOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'ftrl':\n",
    "        return tf.train.FtrlOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'gd':\n",
    "        return tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'padagrad':\n",
    "        return tf.train.ProximalAdagradOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'pgd':\n",
    "        return tf.train.ProximalGradientDescentOptimizer(learning_rate).minimize(loss)\n",
    "    elif opt_algo == 'rmsprop':\n",
    "        return tf.train.RMSPropOptimizer(learning_rate).minimize(loss)\n",
    "    else:\n",
    "        return tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 工具函数\n",
    "# 提示：tf.slice(input_, begin, size, name=None)：按照指定的下标范围抽取连续区域的子集\n",
    "#   tf.gather(params, indices, validate_indices=None, name=None)：按照指定的下标集合从axis=0中抽取子集，适合抽取不连续区域的子集\n",
    "def gather_2d(params, indices):\n",
    "    shape = tf.shape(params)\n",
    "    flat = tf.reshape(params, [-1])\n",
    "    flat_idx = indices[:, 0] * shape[1] + indices[:, 1]\n",
    "    flat_idx = tf.reshape(flat_idx, [-1])\n",
    "    return tf.gather(flat, flat_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gather_3d(params, indices):\n",
    "    shape = tf.shape(params)\n",
    "    flat = tf.reshape(params, [-1])\n",
    "    flat_idx = indices[:, 0] * shape[1] * shape[2] + indices[:, 1] * shape[2] + indices[:, 2]\n",
    "    flat_idx = tf.reshape(flat_idx, [-1])\n",
    "    return tf.gather(flat, flat_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def gather_4d(params, indices):\n",
    "    shape = tf.shape(params)\n",
    "    flat = tf.reshape(params, [-1])\n",
    "    flat_idx = indices[:, 0] * shape[1] * shape[2] * shape[3] + \\\n",
    "               indices[:, 1] * shape[2] * shape[3] + indices[:, 2] * shape[3] + indices[:, 3]\n",
    "    flat_idx = tf.reshape(flat_idx, [-1])\n",
    "    return tf.gather(flat, flat_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 池化2d\n",
    "def max_pool_2d(params, k):\n",
    "    _, indices = tf.nn.top_k(params, k, sorted=False)\n",
    "    shape = tf.shape(indices)\n",
    "    r1 = tf.reshape(tf.range(shape[0]), [-1, 1])\n",
    "    r1 = tf.tile(r1, [1, k])\n",
    "    r1 = tf.reshape(r1, [-1, 1])\n",
    "    indices = tf.concat([r1, tf.reshape(indices, [-1, 1])], 1)\n",
    "    return tf.reshape(gather_2d(params, indices), [-1, k])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 池化3d\n",
    "def max_pool_3d(params, k):\n",
    "    _, indices = tf.nn.top_k(params, k, sorted=False)\n",
    "    shape = tf.shape(indices)\n",
    "    r1 = tf.reshape(tf.range(shape[0]), [-1, 1])\n",
    "    r2 = tf.reshape(tf.range(shape[1]), [-1, 1])\n",
    "    r1 = tf.tile(r1, [1, k * shape[1]])\n",
    "    r2 = tf.tile(r2, [1, k])\n",
    "    r1 = tf.reshape(r1, [-1, 1])\n",
    "    r2 = tf.tile(tf.reshape(r2, [-1, 1]), [shape[0], 1])\n",
    "    indices = tf.concat([r1, r2, tf.reshape(indices, [-1, 1])], 1)\n",
    "    return tf.reshape(gather_3d(params, indices), [-1, shape[1], k])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 池化4d\n",
    "def max_pool_4d(params, k):\n",
    "    _, indices = tf.nn.top_k(params, k, sorted=False)\n",
    "    shape = tf.shape(indices)\n",
    "    r1 = tf.reshape(tf.range(shape[0]), [-1, 1])\n",
    "    r2 = tf.reshape(tf.range(shape[1]), [-1, 1])\n",
    "    r3 = tf.reshape(tf.range(shape[2]), [-1, 1])\n",
    "    r1 = tf.tile(r1, [1, shape[1] * shape[2] * k])\n",
    "    r2 = tf.tile(r2, [1, shape[2] * k])\n",
    "    r3 = tf.tile(r3, [1, k])\n",
    "    r1 = tf.reshape(r1, [-1, 1])\n",
    "    r2 = tf.tile(tf.reshape(r2, [-1, 1]), [shape[0], 1])\n",
    "    r3 = tf.tile(tf.reshape(r3, [-1, 1]), [shape[0] * shape[1], 1])\n",
    "    indices = tf.concat([r1, r2, r3, tf.reshape(indices, [-1, 1])], 1)\n",
    "    return tf.reshape(gather_4d(params, indices), [-1, shape[1], shape[2], k])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义不同的模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 定义基类模型\n",
    "dtype = DTYPE\n",
    "class Model:\n",
    "    def __init__(self):\n",
    "        self.sess = None\n",
    "        self.X = None\n",
    "        self.y = None\n",
    "        self.layer_keeps = None\n",
    "        self.vars = None\n",
    "        self.keep_prob_train = None\n",
    "        self.keep_prob_test = None\n",
    "\n",
    "    # run model\n",
    "    def run(self, fetches, X=None, y=None, mode='train'):\n",
    "            # 通过feed_dict传入数据\n",
    "            feed_dict = {}\n",
    "            if type(self.X) is list:\n",
    "                for i in range(len(X)):\n",
    "                    feed_dict[self.X[i]] = X[i]\n",
    "            else:\n",
    "                feed_dict[self.X] = X\n",
    "            if y is not None:\n",
    "                feed_dict[self.y] = y\n",
    "            if self.layer_keeps is not None:\n",
    "                if mode == 'train':\n",
    "                    feed_dict[self.layer_keeps] = self.keep_prob_train\n",
    "                elif mode == 'test':\n",
    "                    feed_dict[self.layer_keeps] = self.keep_prob_test\n",
    "            #通过session.run去执行op\n",
    "            return self.sess.run(fetches, feed_dict)\n",
    "\n",
    "    # 模型参数持久化\n",
    "    def dump(self, model_path):\n",
    "        var_map = {}\n",
    "        for name, var in self.vars.iteritems():\n",
    "            var_map[name] = self.run(var)\n",
    "        pkl.dump(var_map, open(model_path, 'wb'))\n",
    "        print('model dumped at', model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.LR逻辑回归\n",
    "![](https://pic3.zhimg.com/80/v2-09c0c9a25fa46886f92404fef41bbb82_hd.jpg)\n",
    "输入输出:{X,y}<br>\n",
    "映射函数f(x)：单层单节点的“DNN”, 宽而不深，sigmoid(wx+b)输出概率，需要大量的人工特征工程，非线性来源于特征处理<br>\n",
    "损失函数：logloss/... + L1/L2/...<br>\n",
    "优化方法：sgd/...<br>\n",
    "评估：logloss/auc/...<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class LR(Model):\n",
    "    def __init__(self, input_dim=None, output_dim=1, init_path=None, opt_algo='gd', learning_rate=1e-2, l2_weight=0,\n",
    "                 random_seed=None):\n",
    "        Model.__init__(self)\n",
    "        # 声明参数\n",
    "        init_vars = [('w', [input_dim, output_dim], 'xavier', dtype),\n",
    "                     ('b', [output_dim], 'zero', dtype)]\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            if random_seed is not None:\n",
    "                tf.set_random_seed(random_seed)\n",
    "            # 用稀疏的placeholder\n",
    "            self.X = tf.sparse_placeholder(dtype)\n",
    "            self.y = tf.placeholder(dtype)\n",
    "            # init参数\n",
    "            self.vars = init_var_map(init_vars, init_path)\n",
    "\n",
    "            w = self.vars['w']\n",
    "            b = self.vars['b']\n",
    "            # sigmoid(wx+b)\n",
    "            xw = tf.sparse_tensor_dense_matmul(self.X, w)\n",
    "            logits = tf.reshape(xw + b, [-1])\n",
    "            self.y_prob = tf.sigmoid(logits)\n",
    "\n",
    "            self.loss = tf.reduce_mean(\n",
    "                tf.nn.sigmoid_cross_entropy_with_logits(labels=self.y, logits=logits)) + \\\n",
    "                        l2_weight * tf.nn.l2_loss(xw)\n",
    "            self.optimizer = get_optimizer(opt_algo, learning_rate, self.loss)\n",
    "            # GPU设定\n",
    "            config = tf.ConfigProto()\n",
    "            config.gpu_options.allow_growth = True\n",
    "            self.sess = tf.Session(config=config)\n",
    "            # 初始化图里的参数\n",
    "            tf.global_variables_initializer().run(session=self.sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "read finish\n",
      "train data size: (1742104, 491713)\n",
      "test data size: (300928, 491713)\n",
      "{'l2_weight': 0, 'learning_rate': 0.1, 'random_seed': 0, 'input_dim': 491713, 'opt_algo': 'gd'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  2% (43 of 1702) |                         | Elapsed Time: 0:00:00 ETA: 0:00:07"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tloss (with l2 norm):0.013310\ttrain-auc: 0.571926\teval-auc: 0.680297\n",
      "[1]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tloss (with l2 norm):0.006428\ttrain-auc: 0.591210\teval-auc: 0.697057\n",
      "[2]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\tloss (with l2 norm):0.006289\ttrain-auc: 0.607686\teval-auc: 0.709637\n",
      "[3]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tloss (with l2 norm):0.006240\ttrain-auc: 0.621519\teval-auc: 0.717216\n",
      "[4]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tloss (with l2 norm):0.006211\ttrain-auc: 0.633832\teval-auc: 0.723307\n",
      "[5]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tloss (with l2 norm):0.006188\ttrain-auc: 0.644035\teval-auc: 0.727950\n",
      "[6]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6]\tloss (with l2 norm):0.006170\ttrain-auc: 0.652786\teval-auc: 0.731563\n",
      "[7]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7]\tloss (with l2 norm):0.006154\ttrain-auc: 0.659413\teval-auc: 0.734468\n",
      "[8]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tloss (with l2 norm):0.006141\ttrain-auc: 0.664736\teval-auc: 0.736973\n",
      "[9]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9]\tloss (with l2 norm):0.006129\ttrain-auc: 0.669460\teval-auc: 0.738954\n",
      "[10]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10]\tloss (with l2 norm):0.006118\ttrain-auc: 0.673616\teval-auc: 0.740694\n",
      "[11]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\tloss (with l2 norm):0.006108\ttrain-auc: 0.677262\teval-auc: 0.742230\n",
      "[12]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12]\tloss (with l2 norm):0.006100\ttrain-auc: 0.680487\teval-auc: 0.743845\n",
      "[13]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13]\tloss (with l2 norm):0.006091\ttrain-auc: 0.683437\teval-auc: 0.745453\n",
      "[14]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14]\tloss (with l2 norm):0.006084\ttrain-auc: 0.686213\teval-auc: 0.747034\n",
      "[15]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15]\tloss (with l2 norm):0.006077\ttrain-auc: 0.688810\teval-auc: 0.748585\n",
      "[16]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16]\tloss (with l2 norm):0.006070\ttrain-auc: 0.691280\teval-auc: 0.750020\n",
      "[17]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17]\tloss (with l2 norm):0.006064\ttrain-auc: 0.693641\teval-auc: 0.751441\n",
      "[18]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tloss (with l2 norm):0.006058\ttrain-auc: 0.695897\teval-auc: 0.752858\n",
      "[19]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19]\tloss (with l2 norm):0.006052\ttrain-auc: 0.698024\teval-auc: 0.754320\n",
      "[20]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20]\tloss (with l2 norm):0.006047\ttrain-auc: 0.700048\teval-auc: 0.755676\n",
      "[21]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21]\tloss (with l2 norm):0.006042\ttrain-auc: 0.701964\teval-auc: 0.756917\n",
      "[22]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:03 Time: 0:00:03\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  1% (22 of 1702) |                        | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22]\tloss (with l2 norm):0.006037\ttrain-auc: 0.703786\teval-auc: 0.758051\n",
      "[23]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 43% (745 of 1702) |##########              | Elapsed Time: 0:00:03 ETA: 0:00:04"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import progressbar\n",
    "train_file = './data/train.txt'\n",
    "test_file = './data/test.txt'\n",
    "\n",
    "input_dim = INPUT_DIM\n",
    "\n",
    "# 读取数据\n",
    "#train_data = read_data(train_file)\n",
    "#test_data = read_data(test_file)\n",
    "train_data = pkl.load(open('./data/train.pkl', 'rb'))\n",
    "#train_data = shuffle(train_data)\n",
    "test_data = pkl.load(open('./data/test.pkl', 'rb'))\n",
    "# pkl.dump(train_data, open('./data/train.pkl', 'wb'))\n",
    "# pkl.dump(test_data, open('./data/test.pkl', 'wb'))\n",
    "\n",
    "# 输出数据信息维度\n",
    "if train_data[1].ndim > 1:\n",
    "    print('label must be 1-dim')\n",
    "    exit(0)\n",
    "print('read finish')\n",
    "print('train data size:', train_data[0].shape)\n",
    "print('test data size:', test_data[0].shape)\n",
    "\n",
    "# 训练集与测试集\n",
    "train_size = train_data[0].shape[0]\n",
    "test_size = test_data[0].shape[0]\n",
    "num_feas = len(FIELD_SIZES)\n",
    "\n",
    "# 超参数设定\n",
    "min_round = 1\n",
    "num_round = 200\n",
    "early_stop_round = 5\n",
    "# train + val\n",
    "batch_size = 1024\n",
    "\n",
    "field_sizes = FIELD_SIZES\n",
    "field_offsets = FIELD_OFFSETS\n",
    "\n",
    "# 逻辑回归参数设定\n",
    "lr_params = {\n",
    "    'input_dim': input_dim,\n",
    "    'opt_algo': 'gd',\n",
    "    'learning_rate': 0.1,\n",
    "    'l2_weight': 0,\n",
    "    'random_seed': 0\n",
    "}\n",
    "print(lr_params)\n",
    "model = LR(**lr_params)\n",
    "print(\"training LR...\")\n",
    "def train(model):\n",
    "    history_score = []\n",
    "    # 执行num_round轮\n",
    "    for i in range(num_round):\n",
    "        # 主要的2个op是优化器和损失\n",
    "        fetches = [model.optimizer, model.loss]\n",
    "        if batch_size > 0:\n",
    "            ls = []\n",
    "            # 进度条工具\n",
    "            bar = progressbar.ProgressBar()\n",
    "            print('[%d]\\ttraining...' % i)\n",
    "            for j in bar(range(int(train_size / batch_size + 1))):\n",
    "                X_i, y_i = slice(train_data, j * batch_size, batch_size)\n",
    "                # 训练，run op\n",
    "                _, l = model.run(fetches, X_i, y_i)\n",
    "                ls.append(l)\n",
    "        elif batch_size == -1:\n",
    "            X_i, y_i = slice(train_data)\n",
    "            _, l = model.run(fetches, X_i, y_i)\n",
    "            ls = [l]\n",
    "        train_preds = []\n",
    "        print('[%d]\\tevaluating...' % i)\n",
    "        bar = progressbar.ProgressBar()\n",
    "        for j in bar(range(int(train_size / 10000 + 1))):\n",
    "            X_i, _ = slice(train_data, j * 10000, 10000)\n",
    "            preds = model.run(model.y_prob, X_i, mode='test')\n",
    "            train_preds.extend(preds)\n",
    "        test_preds = []\n",
    "        bar = progressbar.ProgressBar()\n",
    "        for j in bar(range(int(test_size / 10000 + 1))):\n",
    "            X_i, _ = slice(test_data, j * 10000, 10000)\n",
    "            preds = model.run(model.y_prob, X_i, mode='test')\n",
    "            test_preds.extend(preds)\n",
    "        # 把预估的结果和真实结果拿出来计算auc\n",
    "        train_score = roc_auc_score(train_data[1], train_preds)\n",
    "        test_score = roc_auc_score(test_data[1], test_preds)\n",
    "        # 输出auc信息\n",
    "        print('[%d]\\tloss (with l2 norm):%f\\ttrain-auc: %f\\teval-auc: %f' % (i, np.mean(ls), train_score, test_score))\n",
    "        history_score.append(test_score)\n",
    "        # early stopping\n",
    "        if i > min_round and i > early_stop_round:\n",
    "            if np.argmax(history_score) == i - early_stop_round and history_score[-1] - history_score[\n",
    "                        -1 * early_stop_round] < 1e-5:\n",
    "                print('early stop\\nbest iteration:\\n[%d]\\teval-auc: %f' % (\n",
    "                    np.argmax(history_score), np.max(history_score)))\n",
    "                break\n",
    "\n",
    "train(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.FM\n",
    "FM可以视作有二次交叉的LR，为了控制参数量和充分学习，提出了user vector和item vector的概念\n",
    "![](https://pic2.zhimg.com/80/v2-b4941534912e895542a52eda50f39810_hd.jpg)\n",
    "![](https://pic2.zhimg.com/80/v2-098dc05dca6fa4c77d45510cb0951677_hd.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FM(Model):\n",
    "    def __init__(self, input_dim=None, output_dim=1, factor_order=10, init_path=None, opt_algo='gd', learning_rate=1e-2,\n",
    "                 l2_w=0, l2_v=0, random_seed=None):\n",
    "        Model.__init__(self)\n",
    "        # 一次、二次交叉、偏置项\n",
    "        init_vars = [('w', [input_dim, output_dim], 'xavier', dtype),\n",
    "                     ('v', [input_dim, factor_order], 'xavier', dtype),\n",
    "                     ('b', [output_dim], 'zero', dtype)]\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            if random_seed is not None:\n",
    "                tf.set_random_seed(random_seed)\n",
    "            self.X = tf.sparse_placeholder(dtype)\n",
    "            self.y = tf.placeholder(dtype)\n",
    "            self.vars = init_var_map(init_vars, init_path)\n",
    "\n",
    "            w = self.vars['w']\n",
    "            v = self.vars['v']\n",
    "            b = self.vars['b']\n",
    "            \n",
    "            # [(x1+x2+x3)^2 - (x1^2+x2^2+x3^2)]/2\n",
    "            # 先计算所有的交叉项，再减去平方项(自己和自己相乘)\n",
    "            X_square = tf.SparseTensor(self.X.indices, tf.square(self.X.values), tf.to_int64(tf.shape(self.X)))\n",
    "            xv = tf.square(tf.sparse_tensor_dense_matmul(self.X, v))\n",
    "            p = 0.5 * tf.reshape(\n",
    "                tf.reduce_sum(xv - tf.sparse_tensor_dense_matmul(X_square, tf.square(v)), 1),\n",
    "                [-1, output_dim])\n",
    "            xw = tf.sparse_tensor_dense_matmul(self.X, w)\n",
    "            logits = tf.reshape(xw + b + p, [-1])\n",
    "            self.y_prob = tf.sigmoid(logits)\n",
    "\n",
    "            self.loss = tf.reduce_mean(\n",
    "                tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=self.y)) + \\\n",
    "                        l2_w * tf.nn.l2_loss(xw) + \\\n",
    "                        l2_v * tf.nn.l2_loss(xv)\n",
    "            self.optimizer = get_optimizer(opt_algo, learning_rate, self.loss)\n",
    "\n",
    "            #GPU设定\n",
    "            config = tf.ConfigProto()\n",
    "            config.gpu_options.allow_growth = True\n",
    "            self.sess = tf.Session(config=config)\n",
    "            # 图中所有variable初始化\n",
    "            tf.global_variables_initializer().run(session=self.sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "read finish\n",
      "train data size: (1742104, 491713)\n",
      "test data size: (300928, 491713)\n",
      "{'l2_w': 0, 'l2_v': 0, 'factor_order': 10, 'learning_rate': 0.1, 'input_dim': 491713, 'opt_algo': 'gd'}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:37"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:19 Time: 0:00:19\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tloss (with l2 norm):0.013312\ttrain-auc: 0.572517\teval-auc: 0.680232\n",
      "[1]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:19 Time: 0:00:19\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tloss (with l2 norm):0.006431\ttrain-auc: 0.591746\teval-auc: 0.696854\n",
      "[2]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\tloss (with l2 norm):0.006290\ttrain-auc: 0.608052\teval-auc: 0.709254\n",
      "[3]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tloss (with l2 norm):0.006240\ttrain-auc: 0.621882\teval-auc: 0.716983\n",
      "[4]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tloss (with l2 norm):0.006210\ttrain-auc: 0.634064\teval-auc: 0.723087\n",
      "[5]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tloss (with l2 norm):0.006188\ttrain-auc: 0.644236\teval-auc: 0.727615\n",
      "[6]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6]\tloss (with l2 norm):0.006169\ttrain-auc: 0.652855\teval-auc: 0.731220\n",
      "[7]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7]\tloss (with l2 norm):0.006154\ttrain-auc: 0.659409\teval-auc: 0.734181\n",
      "[8]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tloss (with l2 norm):0.006140\ttrain-auc: 0.664734\teval-auc: 0.736699\n",
      "[9]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9]\tloss (with l2 norm):0.006128\ttrain-auc: 0.669438\teval-auc: 0.738620\n",
      "[10]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10]\tloss (with l2 norm):0.006118\ttrain-auc: 0.673561\teval-auc: 0.740343\n",
      "[11]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\tloss (with l2 norm):0.006108\ttrain-auc: 0.677189\teval-auc: 0.741901\n",
      "[12]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12]\tloss (with l2 norm):0.006099\ttrain-auc: 0.680385\teval-auc: 0.743532\n",
      "[13]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13]\tloss (with l2 norm):0.006091\ttrain-auc: 0.683334\teval-auc: 0.745170\n",
      "[14]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14]\tloss (with l2 norm):0.006083\ttrain-auc: 0.686123\teval-auc: 0.746798\n",
      "[15]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15]\tloss (with l2 norm):0.006076\ttrain-auc: 0.688741\teval-auc: 0.748355\n",
      "[16]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16]\tloss (with l2 norm):0.006070\ttrain-auc: 0.691228\teval-auc: 0.749849\n",
      "[17]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17]\tloss (with l2 norm):0.006063\ttrain-auc: 0.693605\teval-auc: 0.751326\n",
      "[18]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  1% (3 of 175) |                          | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tloss (with l2 norm):0.006057\ttrain-auc: 0.695872\teval-auc: 0.752829\n",
      "[19]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19]\tloss (with l2 norm):0.006051\ttrain-auc: 0.698019\teval-auc: 0.754315\n",
      "[20]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20]\tloss (with l2 norm):0.006046\ttrain-auc: 0.700072\teval-auc: 0.755707\n",
      "[21]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21]\tloss (with l2 norm):0.006041\ttrain-auc: 0.702021\teval-auc: 0.756975\n",
      "[22]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22]\tloss (with l2 norm):0.006036\ttrain-auc: 0.703862\teval-auc: 0.758121\n",
      "[23]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[23]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[23]\tloss (with l2 norm):0.006031\ttrain-auc: 0.705590\teval-auc: 0.759207\n",
      "[24]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[24]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[24]\tloss (with l2 norm):0.006026\ttrain-auc: 0.707201\teval-auc: 0.760243\n",
      "[25]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25]\tloss (with l2 norm):0.006022\ttrain-auc: 0.708724\teval-auc: 0.761232\n",
      "[26]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[26]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[26]\tloss (with l2 norm):0.006017\ttrain-auc: 0.710174\teval-auc: 0.762215\n",
      "[27]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[27]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[27]\tloss (with l2 norm):0.006013\ttrain-auc: 0.711557\teval-auc: 0.763158\n",
      "[28]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28]\tloss (with l2 norm):0.006009\ttrain-auc: 0.712884\teval-auc: 0.764041\n",
      "[29]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29]\tloss (with l2 norm):0.006005\ttrain-auc: 0.714163\teval-auc: 0.764895\n",
      "[30]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[30]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[30]\tloss (with l2 norm):0.006001\ttrain-auc: 0.715414\teval-auc: 0.765709\n",
      "[31]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[31]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[31]\tloss (with l2 norm):0.005997\ttrain-auc: 0.716624\teval-auc: 0.766517\n",
      "[32]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32]\tloss (with l2 norm):0.005993\ttrain-auc: 0.717808\teval-auc: 0.767284\n",
      "[33]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33]\tloss (with l2 norm):0.005989\ttrain-auc: 0.718957\teval-auc: 0.768054\n",
      "[34]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[34]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[34]\tloss (with l2 norm):0.005986\ttrain-auc: 0.720085\teval-auc: 0.768790\n",
      "[35]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[35]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[35]\tloss (with l2 norm):0.005982\ttrain-auc: 0.721192\teval-auc: 0.769500\n",
      "[36]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[36]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[36]\tloss (with l2 norm):0.005979\ttrain-auc: 0.722283\teval-auc: 0.770220\n",
      "[37]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[37]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:36"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[37]\tloss (with l2 norm):0.005975\ttrain-auc: 0.723355\teval-auc: 0.770910\n",
      "[38]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[38]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[38]\tloss (with l2 norm):0.005972\ttrain-auc: 0.724409\teval-auc: 0.771589\n",
      "[39]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[39]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[39]\tloss (with l2 norm):0.005968\ttrain-auc: 0.725454\teval-auc: 0.772260\n",
      "[40]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:28"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40]\tloss (with l2 norm):0.005965\ttrain-auc: 0.726482\teval-auc: 0.772911\n",
      "[41]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41]\tloss (with l2 norm):0.005962\ttrain-auc: 0.727501\teval-auc: 0.773554\n",
      "[42]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42]\tloss (with l2 norm):0.005959\ttrain-auc: 0.728506\teval-auc: 0.774188\n",
      "[43]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[43]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[43]\tloss (with l2 norm):0.005955\ttrain-auc: 0.729498\teval-auc: 0.774825\n",
      "[44]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[44]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[44]\tloss (with l2 norm):0.005952\ttrain-auc: 0.730480\teval-auc: 0.775452\n",
      "[45]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[45]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[45]\tloss (with l2 norm):0.005949\ttrain-auc: 0.731445\teval-auc: 0.776082\n",
      "[46]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46]\tloss (with l2 norm):0.005946\ttrain-auc: 0.732409\teval-auc: 0.776694\n",
      "[47]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[47]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[47]\tloss (with l2 norm):0.005943\ttrain-auc: 0.733362\teval-auc: 0.777302\n",
      "[48]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[48]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[48]\tloss (with l2 norm):0.005940\ttrain-auc: 0.734307\teval-auc: 0.777906\n",
      "[49]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[49]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[49]\tloss (with l2 norm):0.005937\ttrain-auc: 0.735243\teval-auc: 0.778527\n",
      "[50]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\tloss (with l2 norm):0.005934\ttrain-auc: 0.736174\teval-auc: 0.779165\n",
      "[51]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[51]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[51]\tloss (with l2 norm):0.005931\ttrain-auc: 0.737104\teval-auc: 0.779798\n",
      "[52]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[52]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[52]\tloss (with l2 norm):0.005928\ttrain-auc: 0.738029\teval-auc: 0.780431\n",
      "[53]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[53]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[53]\tloss (with l2 norm):0.005925\ttrain-auc: 0.738954\teval-auc: 0.781075\n",
      "[54]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[54]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[54]\tloss (with l2 norm):0.005922\ttrain-auc: 0.739882\teval-auc: 0.781734\n",
      "[55]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[55]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[55]\tloss (with l2 norm):0.005919\ttrain-auc: 0.740809\teval-auc: 0.782392\n",
      "[56]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[56]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[56]\tloss (with l2 norm):0.005916\ttrain-auc: 0.741742\teval-auc: 0.783065\n",
      "[57]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[57]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[57]\tloss (with l2 norm):0.005913\ttrain-auc: 0.742674\teval-auc: 0.783734\n",
      "[58]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[58]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[58]\tloss (with l2 norm):0.005910\ttrain-auc: 0.743603\teval-auc: 0.784390\n",
      "[59]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[59]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[59]\tloss (with l2 norm):0.005908\ttrain-auc: 0.744541\teval-auc: 0.785050\n",
      "[60]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[60]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[60]\tloss (with l2 norm):0.005905\ttrain-auc: 0.745479\teval-auc: 0.785693\n",
      "[61]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[61]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[61]\tloss (with l2 norm):0.005902\ttrain-auc: 0.746413\teval-auc: 0.786332\n",
      "[62]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[62]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[62]\tloss (with l2 norm):0.005899\ttrain-auc: 0.747347\teval-auc: 0.786966\n",
      "[63]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[63]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[63]\tloss (with l2 norm):0.005896\ttrain-auc: 0.748281\teval-auc: 0.787589\n",
      "[64]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[64]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[64]\tloss (with l2 norm):0.005894\ttrain-auc: 0.749219\teval-auc: 0.788193\n",
      "[65]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[65]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[65]\tloss (with l2 norm):0.005891\ttrain-auc: 0.750158\teval-auc: 0.788766\n",
      "[66]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[66]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[66]\tloss (with l2 norm):0.005888\ttrain-auc: 0.751093\teval-auc: 0.789330\n",
      "[67]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[67]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[67]\tloss (with l2 norm):0.005885\ttrain-auc: 0.752032\teval-auc: 0.789876\n",
      "[68]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[68]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[68]\tloss (with l2 norm):0.005883\ttrain-auc: 0.752964\teval-auc: 0.790419\n",
      "[69]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[69]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[69]\tloss (with l2 norm):0.005880\ttrain-auc: 0.753902\teval-auc: 0.790936\n",
      "[70]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[70]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[70]\tloss (with l2 norm):0.005877\ttrain-auc: 0.754838\teval-auc: 0.791411\n",
      "[71]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[71]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[71]\tloss (with l2 norm):0.005875\ttrain-auc: 0.755772\teval-auc: 0.791876\n",
      "[72]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[72]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[72]\tloss (with l2 norm):0.005872\ttrain-auc: 0.756704\teval-auc: 0.792331\n",
      "[73]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[73]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[73]\tloss (with l2 norm):0.005870\ttrain-auc: 0.757638\teval-auc: 0.792788\n",
      "[74]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[74]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[74]\tloss (with l2 norm):0.005867\ttrain-auc: 0.758559\teval-auc: 0.793237\n",
      "[75]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[75]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[75]\tloss (with l2 norm):0.005864\ttrain-auc: 0.759475\teval-auc: 0.793674\n",
      "[76]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[76]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[76]\tloss (with l2 norm):0.005862\ttrain-auc: 0.760380\teval-auc: 0.794104\n",
      "[77]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[77]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[77]\tloss (with l2 norm):0.005859\ttrain-auc: 0.761283\teval-auc: 0.794523\n",
      "[78]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[78]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[78]\tloss (with l2 norm):0.005857\ttrain-auc: 0.762179\teval-auc: 0.794943\n",
      "[79]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[79]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[79]\tloss (with l2 norm):0.005854\ttrain-auc: 0.763071\teval-auc: 0.795352\n",
      "[80]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[80]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[80]\tloss (with l2 norm):0.005852\ttrain-auc: 0.763952\teval-auc: 0.795765\n",
      "[81]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[81]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[81]\tloss (with l2 norm):0.005849\ttrain-auc: 0.764824\teval-auc: 0.796156\n",
      "[82]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[82]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[82]\tloss (with l2 norm):0.005847\ttrain-auc: 0.765688\teval-auc: 0.796567\n",
      "[83]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[83]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[83]\tloss (with l2 norm):0.005845\ttrain-auc: 0.766541\teval-auc: 0.796970\n",
      "[84]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[84]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[84]\tloss (with l2 norm):0.005842\ttrain-auc: 0.767385\teval-auc: 0.797374\n",
      "[85]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[85]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[85]\tloss (with l2 norm):0.005840\ttrain-auc: 0.768219\teval-auc: 0.797772\n",
      "[86]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[86]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[86]\tloss (with l2 norm):0.005837\ttrain-auc: 0.769043\teval-auc: 0.798171\n",
      "[87]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[87]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[87]\tloss (with l2 norm):0.005835\ttrain-auc: 0.769855\teval-auc: 0.798564\n",
      "[88]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[88]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[88]\tloss (with l2 norm):0.005833\ttrain-auc: 0.770657\teval-auc: 0.798941\n",
      "[89]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[89]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[89]\tloss (with l2 norm):0.005830\ttrain-auc: 0.771454\teval-auc: 0.799306\n",
      "[90]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[90]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[90]\tloss (with l2 norm):0.005828\ttrain-auc: 0.772240\teval-auc: 0.799669\n",
      "[91]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[91]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[91]\tloss (with l2 norm):0.005826\ttrain-auc: 0.773015\teval-auc: 0.800018\n",
      "[92]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[92]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:01 Time: 0:00:01\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[92]\tloss (with l2 norm):0.005823\ttrain-auc: 0.773777\teval-auc: 0.800352\n",
      "[93]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[93]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[93]\tloss (with l2 norm):0.005821\ttrain-auc: 0.774526\teval-auc: 0.800686\n",
      "[94]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[94]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[94]\tloss (with l2 norm):0.005819\ttrain-auc: 0.775270\teval-auc: 0.801004\n",
      "[95]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[95]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[95]\tloss (with l2 norm):0.005817\ttrain-auc: 0.776005\teval-auc: 0.801311\n",
      "[96]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[96]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[96]\tloss (with l2 norm):0.005814\ttrain-auc: 0.776721\teval-auc: 0.801615\n",
      "[97]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[97]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[97]\tloss (with l2 norm):0.005812\ttrain-auc: 0.777430\teval-auc: 0.801900\n",
      "[98]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[98]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[98]\tloss (with l2 norm):0.005810\ttrain-auc: 0.778128\teval-auc: 0.802164\n",
      "[99]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[99]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[99]\tloss (with l2 norm):0.005808\ttrain-auc: 0.778816\teval-auc: 0.802430\n",
      "[100]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[100]\tloss (with l2 norm):0.005806\ttrain-auc: 0.779492\teval-auc: 0.802681\n",
      "[101]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[101]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[101]\tloss (with l2 norm):0.005803\ttrain-auc: 0.780156\teval-auc: 0.802917\n",
      "[102]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[102]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[102]\tloss (with l2 norm):0.005801\ttrain-auc: 0.780813\teval-auc: 0.803150\n",
      "[103]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[103]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[103]\tloss (with l2 norm):0.005799\ttrain-auc: 0.781461\teval-auc: 0.803374\n",
      "[104]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[104]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[104]\tloss (with l2 norm):0.005797\ttrain-auc: 0.782095\teval-auc: 0.803589\n",
      "[105]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  1% (3 of 175) |                          | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[105]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[105]\tloss (with l2 norm):0.005795\ttrain-auc: 0.782722\teval-auc: 0.803793\n",
      "[106]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[106]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[106]\tloss (with l2 norm):0.005793\ttrain-auc: 0.783339\teval-auc: 0.803982\n",
      "[107]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[107]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[107]\tloss (with l2 norm):0.005790\ttrain-auc: 0.783942\teval-auc: 0.804173\n",
      "[108]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[108]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[108]\tloss (with l2 norm):0.005788\ttrain-auc: 0.784537\teval-auc: 0.804346\n",
      "[109]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[109]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[109]\tloss (with l2 norm):0.005786\ttrain-auc: 0.785123\teval-auc: 0.804512\n",
      "[110]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[110]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[110]\tloss (with l2 norm):0.005784\ttrain-auc: 0.785701\teval-auc: 0.804663\n",
      "[111]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[111]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[111]\tloss (with l2 norm):0.005782\ttrain-auc: 0.786269\teval-auc: 0.804820\n",
      "[112]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[112]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[112]\tloss (with l2 norm):0.005780\ttrain-auc: 0.786826\teval-auc: 0.804961\n",
      "[113]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[113]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[113]\tloss (with l2 norm):0.005778\ttrain-auc: 0.787374\teval-auc: 0.805094\n",
      "[114]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[114]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[114]\tloss (with l2 norm):0.005775\ttrain-auc: 0.787915\teval-auc: 0.805220\n",
      "[115]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[115]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[115]\tloss (with l2 norm):0.005773\ttrain-auc: 0.788450\teval-auc: 0.805341\n",
      "[116]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[116]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[116]\tloss (with l2 norm):0.005771\ttrain-auc: 0.788977\teval-auc: 0.805453\n",
      "[117]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[117]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[117]\tloss (with l2 norm):0.005769\ttrain-auc: 0.789497\teval-auc: 0.805562\n",
      "[118]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[118]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[118]\tloss (with l2 norm):0.005767\ttrain-auc: 0.790007\teval-auc: 0.805666\n",
      "[119]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[119]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[119]\tloss (with l2 norm):0.005765\ttrain-auc: 0.790509\teval-auc: 0.805770\n",
      "[120]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[120]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[120]\tloss (with l2 norm):0.005763\ttrain-auc: 0.791010\teval-auc: 0.805870\n",
      "[121]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[121]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[121]\tloss (with l2 norm):0.005761\ttrain-auc: 0.791504\teval-auc: 0.805965\n",
      "[122]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[122]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[122]\tloss (with l2 norm):0.005759\ttrain-auc: 0.791990\teval-auc: 0.806058\n",
      "[123]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[123]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[123]\tloss (with l2 norm):0.005757\ttrain-auc: 0.792471\teval-auc: 0.806146\n",
      "[124]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[124]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[124]\tloss (with l2 norm):0.005755\ttrain-auc: 0.792947\teval-auc: 0.806231\n",
      "[125]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[125]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[125]\tloss (with l2 norm):0.005753\ttrain-auc: 0.793419\teval-auc: 0.806313\n",
      "[126]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[126]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[126]\tloss (with l2 norm):0.005750\ttrain-auc: 0.793884\teval-auc: 0.806395\n",
      "[127]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[127]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[127]\tloss (with l2 norm):0.005748\ttrain-auc: 0.794346\teval-auc: 0.806473\n",
      "[128]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[128]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[128]\tloss (with l2 norm):0.005746\ttrain-auc: 0.794803\teval-auc: 0.806557\n",
      "[129]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[129]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[129]\tloss (with l2 norm):0.005744\ttrain-auc: 0.795253\teval-auc: 0.806634\n",
      "[130]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[130]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[130]\tloss (with l2 norm):0.005742\ttrain-auc: 0.795701\teval-auc: 0.806705\n",
      "[131]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[131]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[131]\tloss (with l2 norm):0.005740\ttrain-auc: 0.796146\teval-auc: 0.806780\n",
      "[132]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[132]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[132]\tloss (with l2 norm):0.005738\ttrain-auc: 0.796586\teval-auc: 0.806855\n",
      "[133]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[133]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[133]\tloss (with l2 norm):0.005736\ttrain-auc: 0.797025\teval-auc: 0.806926\n",
      "[134]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[134]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[134]\tloss (with l2 norm):0.005734\ttrain-auc: 0.797458\teval-auc: 0.806991\n",
      "[135]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[135]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[135]\tloss (with l2 norm):0.005732\ttrain-auc: 0.797890\teval-auc: 0.807056\n",
      "[136]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[136]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[136]\tloss (with l2 norm):0.005730\ttrain-auc: 0.798321\teval-auc: 0.807115\n",
      "[137]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[137]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[137]\tloss (with l2 norm):0.005728\ttrain-auc: 0.798748\teval-auc: 0.807189\n",
      "[138]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[138]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[138]\tloss (with l2 norm):0.005726\ttrain-auc: 0.799173\teval-auc: 0.807235\n",
      "[139]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[139]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:35"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[139]\tloss (with l2 norm):0.005724\ttrain-auc: 0.799597\teval-auc: 0.807287\n",
      "[140]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[140]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[140]\tloss (with l2 norm):0.005722\ttrain-auc: 0.800019\teval-auc: 0.807348\n",
      "[141]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[141]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[141]\tloss (with l2 norm):0.005720\ttrain-auc: 0.800437\teval-auc: 0.807413\n",
      "[142]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[142]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[142]\tloss (with l2 norm):0.005718\ttrain-auc: 0.800857\teval-auc: 0.807476\n",
      "[143]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[143]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[143]\tloss (with l2 norm):0.005716\ttrain-auc: 0.801272\teval-auc: 0.807530\n",
      "[144]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[144]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[144]\tloss (with l2 norm):0.005714\ttrain-auc: 0.801684\teval-auc: 0.807590\n",
      "[145]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[145]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[145]\tloss (with l2 norm):0.005712\ttrain-auc: 0.802093\teval-auc: 0.807659\n",
      "[146]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[146]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[146]\tloss (with l2 norm):0.005710\ttrain-auc: 0.802502\teval-auc: 0.807727\n",
      "[147]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[147]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[147]\tloss (with l2 norm):0.005708\ttrain-auc: 0.802911\teval-auc: 0.807797\n",
      "[148]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[148]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[148]\tloss (with l2 norm):0.005706\ttrain-auc: 0.803312\teval-auc: 0.807856\n",
      "[149]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[149]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[149]\tloss (with l2 norm):0.005704\ttrain-auc: 0.803717\teval-auc: 0.807932\n",
      "[150]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[150]\tloss (with l2 norm):0.005702\ttrain-auc: 0.804122\teval-auc: 0.807991\n",
      "[151]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[151]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[151]\tloss (with l2 norm):0.005700\ttrain-auc: 0.804527\teval-auc: 0.808073\n",
      "[152]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[152]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[152]\tloss (with l2 norm):0.005698\ttrain-auc: 0.804931\teval-auc: 0.808138\n",
      "[153]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[153]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[153]\tloss (with l2 norm):0.005696\ttrain-auc: 0.805334\teval-auc: 0.808200\n",
      "[154]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[154]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[154]\tloss (with l2 norm):0.005694\ttrain-auc: 0.805735\teval-auc: 0.808271\n",
      "[155]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[155]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[155]\tloss (with l2 norm):0.005692\ttrain-auc: 0.806135\teval-auc: 0.808333\n",
      "[156]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[156]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[156]\tloss (with l2 norm):0.005690\ttrain-auc: 0.806537\teval-auc: 0.808393\n",
      "[157]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[157]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[157]\tloss (with l2 norm):0.005688\ttrain-auc: 0.806938\teval-auc: 0.808459\n",
      "[158]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[158]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[158]\tloss (with l2 norm):0.005686\ttrain-auc: 0.807335\teval-auc: 0.808528\n",
      "[159]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[159]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[159]\tloss (with l2 norm):0.005684\ttrain-auc: 0.807732\teval-auc: 0.808602\n",
      "[160]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[160]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[160]\tloss (with l2 norm):0.005682\ttrain-auc: 0.808127\teval-auc: 0.808669\n",
      "[161]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[161]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[161]\tloss (with l2 norm):0.005680\ttrain-auc: 0.808524\teval-auc: 0.808733\n",
      "[162]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[162]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[162]\tloss (with l2 norm):0.005678\ttrain-auc: 0.808921\teval-auc: 0.808788\n",
      "[163]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[163]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[163]\tloss (with l2 norm):0.005676\ttrain-auc: 0.809316\teval-auc: 0.808847\n",
      "[164]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[164]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[164]\tloss (with l2 norm):0.005674\ttrain-auc: 0.809709\teval-auc: 0.808920\n",
      "[165]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[165]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[165]\tloss (with l2 norm):0.005671\ttrain-auc: 0.810099\teval-auc: 0.808986\n",
      "[166]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[166]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:31"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[166]\tloss (with l2 norm):0.005669\ttrain-auc: 0.810490\teval-auc: 0.809052\n",
      "[167]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[167]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[167]\tloss (with l2 norm):0.005667\ttrain-auc: 0.810880\teval-auc: 0.809114\n",
      "[168]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[168]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[168]\tloss (with l2 norm):0.005665\ttrain-auc: 0.811266\teval-auc: 0.809179\n",
      "[169]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[169]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[169]\tloss (with l2 norm):0.005663\ttrain-auc: 0.811655\teval-auc: 0.809238\n",
      "[170]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[170]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[170]\tloss (with l2 norm):0.005661\ttrain-auc: 0.812043\teval-auc: 0.809290\n",
      "[171]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[171]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[171]\tloss (with l2 norm):0.005659\ttrain-auc: 0.812431\teval-auc: 0.809354\n",
      "[172]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[172]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[172]\tloss (with l2 norm):0.005657\ttrain-auc: 0.812818\teval-auc: 0.809402\n",
      "[173]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[173]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[173]\tloss (with l2 norm):0.005655\ttrain-auc: 0.813202\teval-auc: 0.809437\n",
      "[174]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[174]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[174]\tloss (with l2 norm):0.005653\ttrain-auc: 0.813590\teval-auc: 0.809490\n",
      "[175]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[175]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:33"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[175]\tloss (with l2 norm):0.005651\ttrain-auc: 0.813975\teval-auc: 0.809542\n",
      "[176]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[176]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[176]\tloss (with l2 norm):0.005649\ttrain-auc: 0.814360\teval-auc: 0.809587\n",
      "[177]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[177]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[177]\tloss (with l2 norm):0.005647\ttrain-auc: 0.814744\teval-auc: 0.809632\n",
      "[178]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[178]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[178]\tloss (with l2 norm):0.005645\ttrain-auc: 0.815130\teval-auc: 0.809678\n",
      "[179]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[179]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[179]\tloss (with l2 norm):0.005643\ttrain-auc: 0.815516\teval-auc: 0.809731\n",
      "[180]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[180]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[180]\tloss (with l2 norm):0.005641\ttrain-auc: 0.815900\teval-auc: 0.809772\n",
      "[181]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[181]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[181]\tloss (with l2 norm):0.005639\ttrain-auc: 0.816281\teval-auc: 0.809817\n",
      "[182]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[182]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[182]\tloss (with l2 norm):0.005637\ttrain-auc: 0.816659\teval-auc: 0.809850\n",
      "[183]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[183]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[183]\tloss (with l2 norm):0.005635\ttrain-auc: 0.817039\teval-auc: 0.809890\n",
      "[184]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[184]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[184]\tloss (with l2 norm):0.005633\ttrain-auc: 0.817418\teval-auc: 0.809921\n",
      "[185]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[185]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[185]\tloss (with l2 norm):0.005631\ttrain-auc: 0.817795\teval-auc: 0.809954\n",
      "[186]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[186]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[186]\tloss (with l2 norm):0.005629\ttrain-auc: 0.818174\teval-auc: 0.809990\n",
      "[187]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[187]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[187]\tloss (with l2 norm):0.005627\ttrain-auc: 0.818549\teval-auc: 0.810017\n",
      "[188]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[188]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[188]\tloss (with l2 norm):0.005625\ttrain-auc: 0.818925\teval-auc: 0.810045\n",
      "[189]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[189]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[189]\tloss (with l2 norm):0.005623\ttrain-auc: 0.819300\teval-auc: 0.810069\n",
      "[190]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[190]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[190]\tloss (with l2 norm):0.005620\ttrain-auc: 0.819676\teval-auc: 0.810096\n",
      "[191]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[191]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[191]\tloss (with l2 norm):0.005618\ttrain-auc: 0.820051\teval-auc: 0.810119\n",
      "[192]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[192]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:34"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[192]\tloss (with l2 norm):0.005616\ttrain-auc: 0.820427\teval-auc: 0.810137\n",
      "[193]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[193]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[193]\tloss (with l2 norm):0.005614\ttrain-auc: 0.820800\teval-auc: 0.810159\n",
      "[194]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[194]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[194]\tloss (with l2 norm):0.005612\ttrain-auc: 0.821171\teval-auc: 0.810176\n",
      "[195]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:18 Time: 0:00:18\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[195]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[195]\tloss (with l2 norm):0.005610\ttrain-auc: 0.821545\teval-auc: 0.810189\n",
      "[196]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[196]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:29"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[196]\tloss (with l2 norm):0.005608\ttrain-auc: 0.821916\teval-auc: 0.810204\n",
      "[197]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[197]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[197]\tloss (with l2 norm):0.005606\ttrain-auc: 0.822285\teval-auc: 0.810217\n",
      "[198]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:04"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[198]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (6 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:30"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[198]\tloss (with l2 norm):0.005604\ttrain-auc: 0.822655\teval-auc: 0.810227\n",
      "[199]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:17 Time: 0:00:17\n",
      "  2% (5 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[199]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:02 Time: 0:00:02\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[199]\tloss (with l2 norm):0.005602\ttrain-auc: 0.823025\teval-auc: 0.810234\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import progressbar\n",
    "train_file = './data/train.txt'\n",
    "test_file = './data/test.txt'\n",
    "\n",
    "input_dim = INPUT_DIM\n",
    "train_data = pkl.load(open('./data/train.pkl', 'rb'))\n",
    "train_data = shuffle(train_data)\n",
    "test_data = pkl.load(open('./data/test.pkl', 'rb'))\n",
    "\n",
    "if train_data[1].ndim > 1:\n",
    "    print('label must be 1-dim')\n",
    "    exit(0)\n",
    "print('read finish')\n",
    "print('train data size:', train_data[0].shape)\n",
    "print('test data size:', test_data[0].shape)\n",
    "\n",
    "# 训练集与测试集\n",
    "train_size = train_data[0].shape[0]\n",
    "test_size = test_data[0].shape[0]\n",
    "num_feas = len(FIELD_SIZES)\n",
    "\n",
    "# 超参数设定\n",
    "min_round = 1\n",
    "num_round = 200\n",
    "early_stop_round = 5\n",
    "batch_size = 1024\n",
    "\n",
    "field_sizes = FIELD_SIZES\n",
    "field_offsets = FIELD_OFFSETS\n",
    "\n",
    "# FM参数设定\n",
    "fm_params = {\n",
    "    'input_dim': input_dim,\n",
    "    'factor_order': 10,\n",
    "    'opt_algo': 'gd',\n",
    "    'learning_rate': 0.1,\n",
    "    'l2_w': 0,\n",
    "    'l2_v': 0,\n",
    "}\n",
    "print(fm_params)\n",
    "model = FM(**fm_params)\n",
    "print(\"training FM...\")\n",
    "\n",
    "def train(model):\n",
    "    history_score = []\n",
    "    for i in range(num_round):\n",
    "        # 同样是优化器和损失两个op\n",
    "        fetches = [model.optimizer, model.loss]\n",
    "        if batch_size > 0:\n",
    "            ls = []\n",
    "            bar = progressbar.ProgressBar()\n",
    "            print('[%d]\\ttraining...' % i)\n",
    "            for j in bar(range(int(train_size / batch_size + 1))):\n",
    "                X_i, y_i = slice(train_data, j * batch_size, batch_size)\n",
    "                # 训练\n",
    "                _, l = model.run(fetches, X_i, y_i)\n",
    "                ls.append(l)\n",
    "        elif batch_size == -1:\n",
    "            X_i, y_i = slice(train_data)\n",
    "            _, l = model.run(fetches, X_i, y_i)\n",
    "            ls = [l]\n",
    "        train_preds = []\n",
    "        print('[%d]\\tevaluating...' % i)\n",
    "        bar = progressbar.ProgressBar()\n",
    "        for j in bar(range(int(train_size / 10000 + 1))):\n",
    "            X_i, _ = slice(train_data, j * 10000, 10000)\n",
    "            preds = model.run(model.y_prob, X_i, mode='test')\n",
    "            train_preds.extend(preds)\n",
    "        test_preds = []\n",
    "        bar = progressbar.ProgressBar()\n",
    "        for j in bar(range(int(test_size / 10000 + 1))):\n",
    "            X_i, _ = slice(test_data, j * 10000, 10000)\n",
    "            preds = model.run(model.y_prob, X_i, mode='test')\n",
    "            test_preds.extend(preds)\n",
    "        train_score = roc_auc_score(train_data[1], train_preds)\n",
    "        test_score = roc_auc_score(test_data[1], test_preds)\n",
    "        print('[%d]\\tloss (with l2 norm):%f\\ttrain-auc: %f\\teval-auc: %f' % (i, np.mean(ls), train_score, test_score))\n",
    "        history_score.append(test_score)\n",
    "        if i > min_round and i > early_stop_round:\n",
    "            if np.argmax(history_score) == i - early_stop_round and history_score[-1] - history_score[\n",
    "                        -1 * early_stop_round] < 1e-5:\n",
    "                print('early stop\\nbest iteration:\\n[%d]\\teval-auc: %f' % (\n",
    "                    np.argmax(history_score), np.max(history_score)))\n",
    "                break\n",
    "\n",
    "train(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### FNN\n",
    "FNN的考虑是模型的capacity可以进一步提升，以对更复杂的场景建模。<br>\n",
    "FNN可以视作FM + MLP = LR + MF + MLP\n",
    "![](https://pic4.zhimg.com/80/v2-d9ffb1e0ff7707503d4aed085492d3c7_hd.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class FNN(Model):\n",
    "    def __init__(self, field_sizes=None, embed_size=10, layer_sizes=None, layer_acts=None, drop_out=None,\n",
    "                 embed_l2=None, layer_l2=None, init_path=None, opt_algo='gd', learning_rate=1e-2, random_seed=None):\n",
    "        Model.__init__(self)\n",
    "        init_vars = []\n",
    "        num_inputs = len(field_sizes)\n",
    "        for i in range(num_inputs):\n",
    "            init_vars.append(('embed_%d' % i, [field_sizes[i], embed_size], 'xavier', dtype))\n",
    "        node_in = num_inputs * embed_size\n",
    "        for i in range(len(layer_sizes)):\n",
    "            init_vars.append(('w%d' % i, [node_in, layer_sizes[i]], 'xavier', dtype))\n",
    "            init_vars.append(('b%d' % i, [layer_sizes[i]], 'zero', dtype))\n",
    "            node_in = layer_sizes[i]\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            if random_seed is not None:\n",
    "                tf.set_random_seed(random_seed)\n",
    "            self.X = [tf.sparse_placeholder(dtype) for i in range(num_inputs)]\n",
    "            self.y = tf.placeholder(dtype)\n",
    "            self.keep_prob_train = 1 - np.array(drop_out)\n",
    "            self.keep_prob_test = np.ones_like(drop_out)\n",
    "            self.layer_keeps = tf.placeholder(dtype)\n",
    "            self.vars = init_var_map(init_vars, init_path)\n",
    "            w0 = [self.vars['embed_%d' % i] for i in range(num_inputs)]\n",
    "            xw = tf.concat([tf.sparse_tensor_dense_matmul(self.X[i], w0[i]) for i in range(num_inputs)], 1)\n",
    "            l = xw\n",
    "\n",
    "            for i in range(len(layer_sizes)):\n",
    "                wi = self.vars['w%d' % i]\n",
    "                bi = self.vars['b%d' % i]\n",
    "                print(l.shape, wi.shape, bi.shape)\n",
    "                l = tf.nn.dropout(\n",
    "                    activate(\n",
    "                        tf.matmul(l, wi) + bi,\n",
    "                        layer_acts[i]),\n",
    "                    self.layer_keeps[i])\n",
    "\n",
    "            l = tf.squeeze(l)\n",
    "            self.y_prob = tf.sigmoid(l)\n",
    "\n",
    "            self.loss = tf.reduce_mean(\n",
    "                tf.nn.sigmoid_cross_entropy_with_logits(logits=l, labels=self.y))\n",
    "            if layer_l2 is not None:\n",
    "                self.loss += embed_l2 * tf.nn.l2_loss(xw)\n",
    "                for i in range(len(layer_sizes)):\n",
    "                    wi = self.vars['w%d' % i]\n",
    "                    self.loss += layer_l2[i] * tf.nn.l2_loss(wi)\n",
    "            self.optimizer = get_optimizer(opt_algo, learning_rate, self.loss)\n",
    "\n",
    "            config = tf.ConfigProto()\n",
    "            config.gpu_options.allow_growth = True\n",
    "            self.sess = tf.Session(config=config)\n",
    "            tf.global_variables_initializer().run(session=self.sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "read finish\n",
      "train data size: (1742104, 491713)\n",
      "test data size: (300928, 491713)\n",
      "remove empty fields [25, 445852, 36, 371, 4, 11328, 33995, 12, 7, 5, 4, 20, 2, 38, 6, 8]\n",
      "{'field_sizes': [25, 445852, 36, 371, 4, 11328, 33995, 12, 7, 5, 4, 20, 2, 38, 6, 8], 'layer_acts': ['relu', None], 'embed_l2': 0, 'drop_out': [0, 0], 'embed_size': 10, 'random_seed': 0, 'learning_rate': 0.1, 'layer_sizes': [500, 1], 'layer_l2': [0, 0], 'opt_algo': 'gd'}\n",
      "(?, 160) (160, 500) (500,)\n",
      "(?, 500) (500, 1) (1,)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:47"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tloss (with l2 norm):0.009847\ttrain-auc: 0.581307\teval-auc: 0.644375\n",
      "[1]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:45"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\tloss (with l2 norm):0.006281\ttrain-auc: 0.608522\teval-auc: 0.667282\n",
      "[2]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (3 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:57"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\tloss (with l2 norm):0.006206\ttrain-auc: 0.625896\teval-auc: 0.682097\n",
      "[3]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\tloss (with l2 norm):0.006159\ttrain-auc: 0.639077\teval-auc: 0.694792\n",
      "[4]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[4]\tloss (with l2 norm):0.006125\ttrain-auc: 0.650153\teval-auc: 0.706379\n",
      "[5]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\tloss (with l2 norm):0.006099\ttrain-auc: 0.660329\teval-auc: 0.717581\n",
      "[6]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[6]\tloss (with l2 norm):0.006078\ttrain-auc: 0.669520\teval-auc: 0.727669\n",
      "[7]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[7]\tloss (with l2 norm):0.006062\ttrain-auc: 0.677378\teval-auc: 0.736270\n",
      "[8]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[8]\tloss (with l2 norm):0.006048\ttrain-auc: 0.684017\teval-auc: 0.743570\n",
      "[9]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[9]\tloss (with l2 norm):0.006037\ttrain-auc: 0.689573\teval-auc: 0.749456\n",
      "[10]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[10]\tloss (with l2 norm):0.006027\ttrain-auc: 0.694193\teval-auc: 0.753969\n",
      "[11]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11]\tloss (with l2 norm):0.006019\ttrain-auc: 0.697991\teval-auc: 0.757776\n",
      "[12]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:37"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12]\tloss (with l2 norm):0.006012\ttrain-auc: 0.701157\teval-auc: 0.760820\n",
      "[13]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:45"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13]\tloss (with l2 norm):0.006005\ttrain-auc: 0.703853\teval-auc: 0.763331\n",
      "[14]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[14]\tloss (with l2 norm):0.006000\ttrain-auc: 0.706178\teval-auc: 0.765325\n",
      "[15]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[15]\tloss (with l2 norm):0.005995\ttrain-auc: 0.708246\teval-auc: 0.766979\n",
      "[16]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[16]\tloss (with l2 norm):0.005991\ttrain-auc: 0.710113\teval-auc: 0.768357\n",
      "[17]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[17]\tloss (with l2 norm):0.005987\ttrain-auc: 0.711873\teval-auc: 0.769506\n",
      "[18]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                          | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[18]\tloss (with l2 norm):0.005983\ttrain-auc: 0.713545\teval-auc: 0.770486\n",
      "[19]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[19]\tloss (with l2 norm):0.005980\ttrain-auc: 0.715137\teval-auc: 0.771379\n",
      "[20]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[20]\tloss (with l2 norm):0.005977\ttrain-auc: 0.716647\teval-auc: 0.772130\n",
      "[21]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[21]\tloss (with l2 norm):0.005974\ttrain-auc: 0.718079\teval-auc: 0.772788\n",
      "[22]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[22]\tloss (with l2 norm):0.005971\ttrain-auc: 0.719445\teval-auc: 0.773440\n",
      "[23]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[23]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[23]\tloss (with l2 norm):0.005968\ttrain-auc: 0.720751\teval-auc: 0.774004\n",
      "[24]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[24]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[24]\tloss (with l2 norm):0.005966\ttrain-auc: 0.722009\teval-auc: 0.774528\n",
      "[25]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[25]\tloss (with l2 norm):0.005963\ttrain-auc: 0.723223\teval-auc: 0.775038\n",
      "[26]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[26]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[26]\tloss (with l2 norm):0.005961\ttrain-auc: 0.724407\teval-auc: 0.775512\n",
      "[27]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                          | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[27]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:44"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[27]\tloss (with l2 norm):0.005959\ttrain-auc: 0.725545\teval-auc: 0.775912\n",
      "[28]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[28]\tloss (with l2 norm):0.005957\ttrain-auc: 0.726659\teval-auc: 0.776282\n",
      "[29]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[29]\tloss (with l2 norm):0.005955\ttrain-auc: 0.727729\teval-auc: 0.776646\n",
      "[30]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[30]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[30]\tloss (with l2 norm):0.005952\ttrain-auc: 0.728766\teval-auc: 0.777004\n",
      "[31]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[31]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[31]\tloss (with l2 norm):0.005950\ttrain-auc: 0.729771\teval-auc: 0.777317\n",
      "[32]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[32]\tloss (with l2 norm):0.005948\ttrain-auc: 0.730748\teval-auc: 0.777607\n",
      "[33]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[33]\tloss (with l2 norm):0.005946\ttrain-auc: 0.731701\teval-auc: 0.777890\n",
      "[34]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[34]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[34]\tloss (with l2 norm):0.005944\ttrain-auc: 0.732625\teval-auc: 0.778188\n",
      "[35]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[35]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[35]\tloss (with l2 norm):0.005942\ttrain-auc: 0.733536\teval-auc: 0.778467\n",
      "[36]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[36]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[36]\tloss (with l2 norm):0.005940\ttrain-auc: 0.734430\teval-auc: 0.778747\n",
      "[37]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[37]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[37]\tloss (with l2 norm):0.005938\ttrain-auc: 0.735314\teval-auc: 0.779026\n",
      "[38]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[38]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[38]\tloss (with l2 norm):0.005937\ttrain-auc: 0.736179\teval-auc: 0.779294\n",
      "[39]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[39]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[39]\tloss (with l2 norm):0.005935\ttrain-auc: 0.737031\teval-auc: 0.779562\n",
      "[40]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[40]\tloss (with l2 norm):0.005933\ttrain-auc: 0.737870\teval-auc: 0.779827\n",
      "[41]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[41]\tloss (with l2 norm):0.005931\ttrain-auc: 0.738701\teval-auc: 0.780079\n",
      "[42]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[42]\tloss (with l2 norm):0.005929\ttrain-auc: 0.739521\teval-auc: 0.780356\n",
      "[43]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:23 Time: 0:00:23\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[43]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[43]\tloss (with l2 norm):0.005927\ttrain-auc: 0.740340\teval-auc: 0.780629\n",
      "[44]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[44]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[44]\tloss (with l2 norm):0.005925\ttrain-auc: 0.741152\teval-auc: 0.780899\n",
      "[45]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[45]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[45]\tloss (with l2 norm):0.005923\ttrain-auc: 0.741951\teval-auc: 0.781198\n",
      "[46]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                          | Elapsed Time: 0:00:00 ETA:  0:00:00"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[46]\tloss (with l2 norm):0.005921\ttrain-auc: 0.742746\teval-auc: 0.781472\n",
      "[47]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[47]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[47]\tloss (with l2 norm):0.005919\ttrain-auc: 0.743541\teval-auc: 0.781722\n",
      "[48]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[48]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[48]\tloss (with l2 norm):0.005917\ttrain-auc: 0.744337\teval-auc: 0.781993\n",
      "[49]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[49]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[49]\tloss (with l2 norm):0.005915\ttrain-auc: 0.745134\teval-auc: 0.782266\n",
      "[50]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[50]\tloss (with l2 norm):0.005913\ttrain-auc: 0.745921\teval-auc: 0.782548\n",
      "[51]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[51]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[51]\tloss (with l2 norm):0.005911\ttrain-auc: 0.746717\teval-auc: 0.782820\n",
      "[52]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[52]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (5 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:38"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[52]\tloss (with l2 norm):0.005909\ttrain-auc: 0.747517\teval-auc: 0.783118\n",
      "[53]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[53]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:43"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[53]\tloss (with l2 norm):0.005907\ttrain-auc: 0.748325\teval-auc: 0.783410\n",
      "[54]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:05"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[54]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      "100% (31 of 31) |##########################| Elapsed Time: 0:00:00 Time: 0:00:00\n",
      "  0% (4 of 1702) |                          | Elapsed Time: 0:00:00 ETA: 0:00:42"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[54]\tloss (with l2 norm):0.005905\ttrain-auc: 0.749130\teval-auc: 0.783702\n",
      "[55]\ttraining...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (1702 of 1702) |######################| Elapsed Time: 0:00:24 Time: 0:00:24\n",
      "  1% (3 of 175) |                           | Elapsed Time: 0:00:00 ETA: 0:00:06"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[55]\tevaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (175 of 175) |########################| Elapsed Time: 0:00:04 Time: 0:00:04\n",
      " 32% (10 of 31) |########                   | Elapsed Time: 0:00:00 ETA: 0:00:01"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics import roc_auc_score\n",
    "import progressbar\n",
    "train_file = './data/train.txt'\n",
    "test_file = './data/test.txt'\n",
    "\n",
    "input_dim = INPUT_DIM\n",
    "train_data = pkl.load(open('./data/train.pkl', 'rb'))\n",
    "train_data = shuffle(train_data)\n",
    "test_data = pkl.load(open('./data/test.pkl', 'rb'))\n",
    "\n",
    "if train_data[1].ndim > 1:\n",
    "    print('label must be 1-dim')\n",
    "    exit(0)\n",
    "print('read finish')\n",
    "print('train data size:', train_data[0].shape)\n",
    "print('test data size:', test_data[0].shape)\n",
    "\n",
    "train_size = train_data[0].shape[0]\n",
    "test_size = test_data[0].shape[0]\n",
    "num_feas = len(FIELD_SIZES)\n",
    "\n",
    "min_round = 1\n",
    "num_round = 200\n",
    "early_stop_round = 5\n",
    "batch_size = 1024\n",
    "\n",
    "field_sizes = FIELD_SIZES\n",
    "field_offsets = FIELD_OFFSETS\n",
    "\n",
    "train_data = split_data(train_data)\n",
    "test_data = split_data(test_data)\n",
    "tmp = []\n",
    "for x in field_sizes:\n",
    "    if x > 0:\n",
    "        tmp.append(x)\n",
    "field_sizes = tmp\n",
    "print('remove empty fields', field_sizes)\n",
    "    \n",
    "fnn_params = {\n",
    "    'field_sizes': field_sizes,\n",
    "    'embed_size': 10,\n",
    "    'layer_sizes': [500, 1],\n",
    "    'layer_acts': ['relu', None],\n",
    "    'drop_out': [0, 0],\n",
    "    'opt_algo': 'gd',\n",
    "    'learning_rate': 0.1,\n",
    "    'embed_l2': 0,\n",
    "    'layer_l2': [0, 0],\n",
    "    'random_seed': 0\n",
    "}\n",
    "print(fnn_params)\n",
    "model = FNN(**fnn_params)\n",
    "\n",
    "def train(model):\n",
    "    history_score = []\n",
    "    for i in range(num_round):\n",
    "        fetches = [model.optimizer, model.loss]\n",
    "        if batch_size > 0:\n",
    "            ls = []\n",
    "            bar = progressbar.ProgressBar()\n",
    "            print('[%d]\\ttraining...' % i)\n",
    "            for j in bar(range(int(train_size / batch_size + 1))):\n",
    "                X_i, y_i = slice(train_data, j * batch_size, batch_size)\n",
    "                _, l = model.run(fetches, X_i, y_i)\n",
    "                ls.append(l)\n",
    "        elif batch_size == -1:\n",
    "            X_i, y_i = slice(train_data)\n",
    "            _, l = model.run(fetches, X_i, y_i)\n",
    "            ls = [l]\n",
    "        train_preds = []\n",
    "        print('[%d]\\tevaluating...' % i)\n",
    "        bar = progressbar.ProgressBar()\n",
    "        for j in bar(range(int(train_size / 10000 + 1))):\n",
    "            X_i, _ = slice(train_data, j * 10000, 10000)\n",
    "            preds = model.run(model.y_prob, X_i, mode='test')\n",
    "            train_preds.extend(preds)\n",
    "        test_preds = []\n",
    "        bar = progressbar.ProgressBar()\n",
    "        for j in bar(range(int(test_size / 10000 + 1))):\n",
    "            X_i, _ = slice(test_data, j * 10000, 10000)\n",
    "            preds = model.run(model.y_prob, X_i, mode='test')\n",
    "            test_preds.extend(preds)\n",
    "        train_score = roc_auc_score(train_data[1], train_preds)\n",
    "        test_score = roc_auc_score(test_data[1], test_preds)\n",
    "        print('[%d]\\tloss (with l2 norm):%f\\ttrain-auc: %f\\teval-auc: %f' % (i, np.mean(ls), train_score, test_score))\n",
    "        history_score.append(test_score)\n",
    "        if i > min_round and i > early_stop_round:\n",
    "            if np.argmax(history_score) == i - early_stop_round and history_score[-1] - history_score[\n",
    "                        -1 * early_stop_round] < 1e-5:\n",
    "                print('early stop\\nbest iteration:\\n[%d]\\teval-auc: %f' % (\n",
    "                    np.argmax(history_score), np.max(history_score)))\n",
    "                break\n",
    "\n",
    "train(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CCPM\n",
    "reference：[ctr模型汇总](https://zhuanlan.zhihu.com/p/32523455)\n",
    "\n",
    "FM只能学习特征的二阶组合，但CNN能学习更高阶的组合，可学习的阶数和卷积的视野相关。\n",
    "![](https://img-blog.csdn.net/20171211204240715?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvRGFueUhnYw==/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/SouthEast)\n",
    "mbedding层：e1, e2…en是某特定用户被展示的一系列广告。如果在预测广告是否会点击时不考虑历史展示广告的点击情况，则n=1。同时embedding矩阵的具体值是随着模型训练学出来的。Embedding矩阵为S，向量维度为d。\n",
    "\n",
    "卷积层：卷积参数W有d*w个，即对于矩阵S，上图每一列对应一个参数不共享的一维卷积，其视野为w，卷积共有d个，每个输出向量维度为(n+w-1)，输出矩阵维度d*(n+w-1)。因为对于ctr预估而言，矩阵S每一列都对应特定的描述维度，所以需要分别处理，得到的输出矩阵的每一列就都是描述广告特定方面的特征。\n",
    "\n",
    "Pooling层：flexible p-max pooling。\n",
    "![](https://pic1.zhimg.com/80/v2-1c76210b014826e02ebbadf07168715b_hd.jpg)\n",
    "L是模型总卷积层数，n是输入序列长度，pi就是第i层的pooling参数。这样最后一层卷积层都是输出3个最大的元素，长度固定方便后面接全连接层。同时这个指数型的参数，一开始改变比较小，几乎都是n，后面就减少得比较快。这样可以防止在模型浅层的时候就损失太多信息，众所周知深度模型在前面几层最好不要做得太简单，容易损失很多信息。文章还提到p-max pooling输出的几个最大的元素是保序的，可输入时的顺序一致，这点对于保留序列信息是重要的。\n",
    "\n",
    "激活层：tanh\n",
    "\n",
    "最后，\n",
    "![](https://pic3.zhimg.com/80/v2-1c8e3a5f520c66e62312b458b1308d79_hd.jpg)\n",
    "Fij是指低i层的第j个feature map。感觉是不同输入通道的卷积参数也不共享，对应输出是所有输入通道卷积的输出的求和。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class CCPM(Model):\n",
    "    def __init__(self, field_sizes=None, embed_size=10, filter_sizes=None, layer_acts=None, drop_out=None,\n",
    "                 init_path=None, opt_algo='gd', learning_rate=1e-2, random_seed=None):\n",
    "        Model.__init__(self)\n",
    "        init_vars = []\n",
    "        num_inputs = len(field_sizes)\n",
    "        for i in range(num_inputs):\n",
    "            init_vars.append(('embed_%d' % i, [field_sizes[i], embed_size], 'xavier', dtype))\n",
    "        init_vars.append(('f1', [embed_size, filter_sizes[0], 1, 2], 'xavier', dtype))\n",
    "        init_vars.append(('f2', [embed_size, filter_sizes[1], 2, 2], 'xavier', dtype))\n",
    "        init_vars.append(('w1', [2 * 3 * embed_size, 1], 'xavier', dtype))\n",
    "        init_vars.append(('b1', [1], 'zero', dtype))\n",
    "\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            if random_seed is not None:\n",
    "                tf.set_random_seed(random_seed)\n",
    "            self.X = [tf.sparse_placeholder(dtype) for i in range(num_inputs)]\n",
    "            self.y = tf.placeholder(dtype)\n",
    "            self.keep_prob_train = 1 - np.array(drop_out)\n",
    "            self.keep_prob_test = np.ones_like(drop_out)\n",
    "            self.layer_keeps = tf.placeholder(dtype)\n",
    "            self.vars = init_var_map(init_vars, init_path)\n",
    "            w0 = [self.vars['embed_%d' % i] for i in range(num_inputs)]\n",
    "            xw = tf.concat([tf.sparse_tensor_dense_matmul(self.X[i], w0[i]) for i in range(num_inputs)], 1)\n",
    "            l = xw\n",
    "\n",
    "            l = tf.transpose(tf.reshape(l, [-1, num_inputs, embed_size, 1]), [0, 2, 1, 3])\n",
    "            f1 = self.vars['f1']\n",
    "            l = tf.nn.conv2d(l, f1, [1, 1, 1, 1], 'SAME')\n",
    "            l = tf.transpose(\n",
    "                max_pool_4d(\n",
    "                    tf.transpose(l, [0, 1, 3, 2]),\n",
    "                    int(num_inputs / 2)),\n",
    "                [0, 1, 3, 2])\n",
    "            f2 = self.vars['f2']\n",
    "            l = tf.nn.conv2d(l, f2, [1, 1, 1, 1], 'SAME')\n",
    "            l = tf.transpose(\n",
    "                max_pool_4d(\n",
    "                    tf.transpose(l, [0, 1, 3, 2]), 3),\n",
    "                [0, 1, 3, 2])\n",
    "            l = tf.nn.dropout(\n",
    "                activate(\n",
    "                    tf.reshape(l, [-1, embed_size * 3 * 2]),\n",
    "                    layer_acts[0]),\n",
    "                self.layer_keeps[0])\n",
    "            w1 = self.vars['w1']\n",
    "            b1 = self.vars['b1']\n",
    "            l = tf.matmul(l, w1) + b1\n",
    "\n",
    "            l = tf.squeeze(l)\n",
    "            self.y_prob = tf.sigmoid(l)\n",
    "\n",
    "            self.loss = tf.reduce_mean(\n",
    "                tf.nn.sigmoid_cross_entropy_with_logits(logits=l, labels=self.y))\n",
    "            self.optimizer = get_optimizer(opt_algo, learning_rate, self.loss)\n",
    "\n",
    "            config = tf.ConfigProto()\n",
    "            config.gpu_options.allow_growth = True\n",
    "            self.sess = tf.Session(config=config)\n",
    "            tf.global_variables_initializer().run(session=self.sess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PNN\n",
    "reference:<br>\n",
    "[深度学习在CTR预估中的应用](https://zhuanlan.zhihu.com/p/35484389)\n",
    "\n",
    "可以视作FNN+product layer\n",
    "![](https://yxzf.github.io/images/deeplearning/dnn_ctr/pnn.png)\n",
    "\n",
    "PNN和FNN的主要不同在于除了得到z向量，还增加了一个p向量，即Product向量。Product向量由每个category field的feature vector做inner product 或则 outer product 得到，作者认为这样做有助于特征交叉。另外PNN中Embeding层不再由FM生成，可以在整个网络中训练得到。\n",
    "\n",
    "对比 FNN 网络，PNN的区别在于中间多了一层 Product Layer 层。Product Layer 层由两部分组成，左边z为 embedding 层的线性部分，右边为 embedding 层的特征交叉部分。\n",
    "\n",
    "除了 Product Layer 不同，PNN 和 FNN 的 MLP 结构是一样的。这种 product 思想来源于，在 CTR 预估中，认为特征之间的关系更多是一种 and“且”的关系，而非 add\"加”的关系。例如，性别为男且喜欢游戏的人群，比起性别男和喜欢游戏的人群，前者的组合比后者更能体现特征交叉的意义。\n",
    "\n",
    "根据 product 的方式不同，可以分为 inner product (IPNN) 和 outer product (OPNN)，如下图所示。\n",
    "\n",
    "![](https://pic4.zhimg.com/v2-c30b0f9983345382d31a30d4eed516d3_r.jpg)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PNN1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class PNN1(Model):\n",
    "    def __init__(self, field_sizes=None, embed_size=10, layer_sizes=None, layer_acts=None, drop_out=None,\n",
    "                 embed_l2=None, layer_l2=None, init_path=None, opt_algo='gd', learning_rate=1e-2, random_seed=None):\n",
    "        Model.__init__(self)\n",
    "        init_vars = []\n",
    "        num_inputs = len(field_sizes)\n",
    "        for i in range(num_inputs):\n",
    "            init_vars.append(('embed_%d' % i, [field_sizes[i], embed_size], 'xavier', dtype))\n",
    "        num_pairs = int(num_inputs * (num_inputs - 1) / 2)\n",
    "        node_in = num_inputs * embed_size + num_pairs\n",
    "        # node_in = num_inputs * (embed_size + num_inputs)\n",
    "        for i in range(len(layer_sizes)):\n",
    "            init_vars.append(('w%d' % i, [node_in, layer_sizes[i]], 'xavier', dtype))\n",
    "            init_vars.append(('b%d' % i, [layer_sizes[i]], 'zero', dtype))\n",
    "            node_in = layer_sizes[i]\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            if random_seed is not None:\n",
    "                tf.set_random_seed(random_seed)\n",
    "            self.X = [tf.sparse_placeholder(dtype) for i in range(num_inputs)]\n",
    "            self.y = tf.placeholder(dtype)\n",
    "            self.keep_prob_train = 1 - np.array(drop_out)\n",
    "            self.keep_prob_test = np.ones_like(drop_out)\n",
    "            self.layer_keeps = tf.placeholder(dtype)\n",
    "            self.vars = init_var_map(init_vars, init_path)\n",
    "            w0 = [self.vars['embed_%d' % i] for i in range(num_inputs)]\n",
    "            xw = tf.concat([tf.sparse_tensor_dense_matmul(self.X[i], w0[i]) for i in range(num_inputs)], 1)\n",
    "            xw3d = tf.reshape(xw, [-1, num_inputs, embed_size])\n",
    "\n",
    "            row = []\n",
    "            col = []\n",
    "            for i in range(num_inputs-1):\n",
    "                for j in range(i+1, num_inputs):\n",
    "                    row.append(i)\n",
    "                    col.append(j)\n",
    "            # batch * pair * k\n",
    "            p = tf.transpose(\n",
    "                # pair * batch * k\n",
    "                tf.gather(\n",
    "                    # num * batch * k\n",
    "                    tf.transpose(\n",
    "                        xw3d, [1, 0, 2]),\n",
    "                    row),\n",
    "                [1, 0, 2])\n",
    "            # batch * pair * k\n",
    "            q = tf.transpose(\n",
    "                tf.gather(\n",
    "                    tf.transpose(\n",
    "                        xw3d, [1, 0, 2]),\n",
    "                    col),\n",
    "                [1, 0, 2])\n",
    "            p = tf.reshape(p, [-1, num_pairs, embed_size])\n",
    "            q = tf.reshape(q, [-1, num_pairs, embed_size])\n",
    "            ip = tf.reshape(tf.reduce_sum(p * q, [-1]), [-1, num_pairs])\n",
    "\n",
    "            # simple but redundant\n",
    "            # batch * n * 1 * k, batch * 1 * n * k\n",
    "            # ip = tf.reshape(\n",
    "            #     tf.reduce_sum(\n",
    "            #         tf.expand_dims(xw3d, 2) *\n",
    "            #         tf.expand_dims(xw3d, 1),\n",
    "            #         3),\n",
    "            #     [-1, num_inputs**2])\n",
    "            l = tf.concat([xw, ip], 1)\n",
    "\n",
    "            for i in range(len(layer_sizes)):\n",
    "                wi = self.vars['w%d' % i]\n",
    "                bi = self.vars['b%d' % i]\n",
    "                l = tf.nn.dropout(\n",
    "                    activate(\n",
    "                        tf.matmul(l, wi) + bi,\n",
    "                        layer_acts[i]),\n",
    "                    self.layer_keeps[i])\n",
    "\n",
    "            l = tf.squeeze(l)\n",
    "            self.y_prob = tf.sigmoid(l)\n",
    "\n",
    "            self.loss = tf.reduce_mean(\n",
    "                tf.nn.sigmoid_cross_entropy_with_logits(logits=l, labels=self.y))\n",
    "            if layer_l2 is not None:\n",
    "                self.loss += embed_l2 * tf.nn.l2_loss(xw)\n",
    "                for i in range(len(layer_sizes)):\n",
    "                    wi = self.vars['w%d' % i]\n",
    "                    self.loss += layer_l2[i] * tf.nn.l2_loss(wi)\n",
    "            self.optimizer = get_optimizer(opt_algo, learning_rate, self.loss)\n",
    "\n",
    "            config = tf.ConfigProto()\n",
    "            config.gpu_options.allow_growth = True\n",
    "            self.sess = tf.Session(config=config)\n",
    "            tf.global_variables_initializer().run(session=self.sess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### PNN2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class PNN2(Model):\n",
    "    def __init__(self, field_sizes=None, embed_size=10, layer_sizes=None, layer_acts=None, drop_out=None,\n",
    "                 embed_l2=None, layer_l2=None, init_path=None, opt_algo='gd', learning_rate=1e-2, random_seed=None,\n",
    "                 layer_norm=True):\n",
    "        Model.__init__(self)\n",
    "        init_vars = []\n",
    "        num_inputs = len(field_sizes)\n",
    "        for i in range(num_inputs):\n",
    "            init_vars.append(('embed_%d' % i, [field_sizes[i], embed_size], 'xavier', dtype))\n",
    "        num_pairs = int(num_inputs * (num_inputs - 1) / 2)\n",
    "        node_in = num_inputs * embed_size + num_pairs\n",
    "        init_vars.append(('kernel', [embed_size, num_pairs, embed_size], 'xavier', dtype))\n",
    "        for i in range(len(layer_sizes)):\n",
    "            init_vars.append(('w%d' % i, [node_in, layer_sizes[i]], 'xavier', dtype))\n",
    "            init_vars.append(('b%d' % i, [layer_sizes[i]], 'zero',  dtype))\n",
    "            node_in = layer_sizes[i]\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            if random_seed is not None:\n",
    "                tf.set_random_seed(random_seed)\n",
    "            self.X = [tf.sparse_placeholder(dtype) for i in range(num_inputs)]\n",
    "            self.y = tf.placeholder(dtype)\n",
    "            self.keep_prob_train = 1 - np.array(drop_out)\n",
    "            self.keep_prob_test = np.ones_like(drop_out)\n",
    "            self.layer_keeps = tf.placeholder(dtype)\n",
    "            self.vars = init_var_map(init_vars, init_path)\n",
    "            w0 = [self.vars['embed_%d' % i] for i in range(num_inputs)]\n",
    "            xw = tf.concat([tf.sparse_tensor_dense_matmul(self.X[i], w0[i]) for i in range(num_inputs)], 1)\n",
    "            xw3d = tf.reshape(xw, [-1, num_inputs, embed_size])\n",
    "\n",
    "            row = []\n",
    "            col = []\n",
    "            for i in range(num_inputs - 1):\n",
    "                for j in range(i + 1, num_inputs):\n",
    "                    row.append(i)\n",
    "                    col.append(j)\n",
    "            # batch * pair * k\n",
    "            p = tf.transpose(\n",
    "                # pair * batch * k\n",
    "                tf.gather(\n",
    "                    # num * batch * k\n",
    "                    tf.transpose(\n",
    "                        xw3d, [1, 0, 2]),\n",
    "                    row),\n",
    "                [1, 0, 2])\n",
    "            # batch * pair * k\n",
    "            q = tf.transpose(\n",
    "                tf.gather(\n",
    "                    tf.transpose(\n",
    "                        xw3d, [1, 0, 2]),\n",
    "                    col),\n",
    "                [1, 0, 2])\n",
    "            # b * p * k\n",
    "            p = tf.reshape(p, [-1, num_pairs, embed_size])\n",
    "            # b * p * k\n",
    "            q = tf.reshape(q, [-1, num_pairs, embed_size])\n",
    "            # k * p * k\n",
    "            k = self.vars['kernel']\n",
    "\n",
    "            # batch * 1 * pair * k\n",
    "            p = tf.expand_dims(p, 1)\n",
    "            # batch * pair\n",
    "            kp = tf.reduce_sum(\n",
    "                # batch * pair * k\n",
    "                tf.multiply(\n",
    "                    # batch * pair * k\n",
    "                    tf.transpose(\n",
    "                        # batch * k * pair\n",
    "                        tf.reduce_sum(\n",
    "                            # batch * k * pair * k\n",
    "                            tf.multiply(\n",
    "                                p, k),\n",
    "                            -1),\n",
    "                        [0, 2, 1]),\n",
    "                    q),\n",
    "                -1)\n",
    "\n",
    "            #\n",
    "            # if layer_norm:\n",
    "            #     # x_mean, x_var = tf.nn.moments(xw, [1], keep_dims=True)\n",
    "            #     # xw = (xw - x_mean) / tf.sqrt(x_var)\n",
    "            #     # x_g = tf.Variable(tf.ones([num_inputs * embed_size]), name='x_g')\n",
    "            #     # x_b = tf.Variable(tf.zeros([num_inputs * embed_size]), name='x_b')\n",
    "            #     # x_g = tf.Print(x_g, [x_g[:10], x_b])\n",
    "            #     # xw = xw * x_g + x_b\n",
    "            #     p_mean, p_var = tf.nn.moments(op, [1], keep_dims=True)\n",
    "            #     op = (op - p_mean) / tf.sqrt(p_var)\n",
    "            #     p_g = tf.Variable(tf.ones([embed_size**2]), name='p_g')\n",
    "            #     p_b = tf.Variable(tf.zeros([embed_size**2]), name='p_b')\n",
    "            #     # p_g = tf.Print(p_g, [p_g[:10], p_b])\n",
    "            #     op = op * p_g + p_b\n",
    "\n",
    "            l = tf.concat([xw, kp], 1)\n",
    "            for i in range(len(layer_sizes)):\n",
    "                wi = self.vars['w%d' % i]\n",
    "                bi = self.vars['b%d' % i]\n",
    "                l = tf.nn.dropout(\n",
    "                    activate(\n",
    "                        tf.matmul(l, wi) + bi,\n",
    "                        layer_acts[i]),\n",
    "                    self.layer_keeps[i])\n",
    "\n",
    "            l = tf.squeeze(l)\n",
    "            self.y_prob = tf.sigmoid(l)\n",
    "\n",
    "            self.loss = tf.reduce_mean(\n",
    "                tf.nn.sigmoid_cross_entropy_with_logits(logits=l, labels=self.y))\n",
    "            if layer_l2 is not None:\n",
    "                self.loss += embed_l2 * tf.nn.l2_loss(xw)#tf.concat(w0, 0))\n",
    "                for i in range(len(layer_sizes)):\n",
    "                    wi = self.vars['w%d' % i]\n",
    "                    self.loss += layer_l2[i] * tf.nn.l2_loss(wi)\n",
    "            self.optimizer = get_optimizer(opt_algo, learning_rate, self.loss)\n",
    "\n",
    "            config = tf.ConfigProto()\n",
    "            config.gpu_options.allow_growth = True\n",
    "            self.sess = tf.Session(config=config)\n",
    "            tf.global_variables_initializer().run(session=self.sess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
